Harness Theory Part 08 | Reliability engineering: retries, fallbacks and side-effect control

Suppose a support agent receives a refund request.

It checks the order, confirms that the policy allows a refund, then calls the payment service. The request leaves the harness, but the connection times out before a response arrives.

What should happen next?

The most dangerous answer is: try again.

The payment provider may never have received the first request. It may also have completed the refund and lost only the response on the way back. If the harness treats a timeout as proof that nothing happened, the second call may create a duplicate refund.

Reliability failures in an agent system are often not reasoning failures. They are failures to answer a more operational set of questions:

What class of failure occurred?
Did the previous operation already create a side effect?
Is it safe to repeat?
Should the system retry, repair, fall back, replan, compensate or stop?
How much time, cost and retry budget remains?
After a worker crash, which state can the next worker trust?

Reliability does not mean adding more retries. It means giving each failure class a bounded, observable and recoverable path.

Failure class answers only half the question

In a chat product, failure may mean that no answer arrived. Once an agent can call tools, mutate state, and start background work, a useful decision needs three dimensions:

Failure domain: model, schema, policy, dependency, state, capacity, or budget.
Effect certainty: definitely not executed, possibly executed, partially complete, confirmed complete, or still unknown.
Recovery eligibility: whether the current contract, deadline, cost, idempotency guarantee, and authority allow another action.

Failure domain	Typical case	Effect certainty	Common direction
Model	Timeout, unavailable provider, truncated output	Usually no external side effect	Bounded retry, compatible fallback, continuation
Schema or validation	Tool arguments fail the contract	Not executed	Repair or regenerate arguments
Permission or policy	Missing authority or policy denial	Not executed	Stop, wait for approval, or choose an allowed route
Transient dependency	Rate limit or short network fault	Depends on the operation	Backoff, jitter, bounded retry, or reconciliation
Permanent business	Insufficient balance or non-refundable order	Usually a known failure	Report the reason; do not retry
State conflict	Version conflict or stale claim	Local commit failed; external state is separate	Reload canonical state, repair, or replan
Unknown outcome	Timeout after submission	A side effect may already exist	Reconcile; do not resend blindly
Partial outcome	Only part of a batch or saga completed	Some effects exist	Forward repair, compensate, or human review
Capacity or overload	Old queue item or unavailable sandbox	Work has not started or is stranded	Admission control, backpressure, or load shedding
Budget exhaustion	Cost, time, step, or retry budget consumed	Depends on the current step	Degrade, checkpoint, hand over, or stop

This taxonomy is not decorative logging. It determines whether the next state transition is safe.

INVALID_ARGUMENTS and RATE_LIMIT may both fail a tool call, but resending the first request unchanged repairs nothing. PERMISSION_DENIED is not a network wobble. The most important separate state is UNKNOWN_OUTCOME: the caller lacks a result while the external world may already have changed.

Error classification and recovery action matrix

Figure 8-1 | A recovery decision combines failure domain, side-effect certainty, retry safety, and remaining budget rather than mapping every error class to one fixed action.

A recovery action is also a controlled state transition

Recovery should not be scattered across tool-wrapper try/except blocks or delegated wholesale to the model.

Common actions include:

Retry: the fault may be transient, the operation is safe to repeat, and budget remains.
Repair: correct schema, arguments, query, or missing input before another attempt.
Fallback: move to a known-compatible model, provider, implementation, or degraded service path.
Replan: the present path is blocked, but another authorised path may complete the task.
Reconcile: inspect external authority because the outcome is unknown.
Forward repair: preserve an established side effect and repair the state that follows it.
Compensate: perform a new corrective action rather than pretending that time reversed.
Quarantine: isolate a poison message, unknown version, or repeatedly failing work unit for bounded review.
Ask a human: approval, commercial judgement, or high-risk ambiguity prevents autonomous recovery.
Stop: policy, permanent error, information loss, or budget forbids continuation.

recovery_policy:
  RATE_LIMIT:
    action: retry
    owner: dependency_adapter
    max_attempts: 3
    backoff: exponential_with_jitter

  INVALID_TOOL_ARGUMENTS:
    action: repair
    owner: agent_controller
    max_repairs: 2

  PERMISSION_DENIED:
    action: stop

  UNKNOWN_SIDE_EFFECT_OUTCOME:
    action: reconcile
    blind_resend: false

  POISON_MESSAGE:
    action: quarantine

  BUDGET_EXCEEDED:
    action: stop

This YAML locates ownership, budget, and outcome for a recovery decision. It does not establish correct durable state, idempotency, external reconciliation, concurrency control, or error mapping.

RunState should retain at least:

failure domain and effect certainty
logical and transport attempts
recovery owner
last delay and Retry-After
whether provider, model, or tool changed
side-effect state and reconciliation cursor
remaining budget
terminal or quarantine reason

Recovery actions can fail too. Compensation, status queries, fallback, and notification need their own attempts, idempotency, and stopping rules. Otherwise the unbounded loop merely moves into the repair path.

Timeouts, retries and backoff solve different problems

These terms often appear together, but they have separate responsibilities.

A timeout bounds waiting

A timeout answers: how long may this operation remain unresolved?

A harness normally needs separate limits for:

model calls
individual tools
the whole workflow or run
approval expiry
queue waiting time

A timeout is an observation made by the caller. It usually means only that no answer arrived within the deadline. It does not prove that the remote operation never started, or that it failed.

A retry trades another attempt for a higher chance of success

Retries suit transient faults such as short network interruptions, rate limits and recoverable provider errors. They add latency, cost and downstream load.

The AWS Builders’ Library describes retries as selfish: the caller increases its own probability of success while adding work to a dependency that may already be overloaded. When several service layers retry independently, traffic can multiply rather than merely increase.¹

A retry policy should therefore define at least:

maximum attempts
total retry duration
which layer owns retry authority
retryable error codes
whether the operation is idempotent
whether the run deadline permits another attempt
whether provider Retry-After guidance applies

Backoff and jitter prevent synchronised retries

Exponential backoff increases the wait between attempts. Jitter adds randomness so that many clients do not fail together, wake together and produce another coordinated surge.

Neither mechanism makes a permanent error retryable, and neither makes a side effect idempotent. Adding jitter to duplicate payment requests merely gives the incident a more varied rhythm.

Budget and retry authority must be visible across layers

An agent can replan, retrieve again, regenerate tool arguments, change models, and rerun verification. Every recovery path spends resources.

Common budgets include:

maximum steps
maximum model and tool calls
logical retry count
transport retry count
token and monetary budget
wall-clock deadline
retrieval depth
context information-loss limit

The runtime must enforce and persist these limits. A prompt saying “try no more than three times” is not a control.

Hidden retries are more dangerous:

the SDK retries three times
the tool wrapper tries three more
the agent loop invokes the tool again
the queue redelivers after a consumer crash

Each layer appears conservative in isolation while the composition produces dozens of calls. A dependable design assigns one retry owner per boundary, or at least charges library retries to one shared attempt ledger.

Downstream calls should also receive the remaining deadline rather than owning timeouts longer than the parent run. When the remaining time cannot accommodate backoff, execution, and verification, another attempt should not be scheduled.

Retry traffic is load. The AWS Builders’ Library notes that a retry improves the caller’s chance of success while adding work to a dependency that may already be overloaded; layered retries amplify failure.¹

Fallback must preserve the contract or declare degradation

The purpose of fallback is to maintain an acceptable service, not to find any provider that responds.

Before switching, check:

tool and structured-output compatibility
context capacity
safety, policy, privacy, and data residency
credentials, region, and network path
quality floor and verification requirement
cost, deadline, and rate limits
whether the new adapter can interpret current RunState and checkpoints
whether the new route changes side-effect or audit semantics

A primary model may support tool calling and a large context while a fallback supports text only. A successful API response does not preserve the original task contract.

Fallback can also mean a degraded service:

live search uses a cache with visible freshness
a high-precision reranker falls back to basic ranking
browser automation becomes a read-only API path
full analysis becomes sampling

The result needs an explicit degraded, partial, or comparable state and another completion check. Verification performed under the stronger path does not automatically survive a reduction in evidence quality.

Fallback also needs anti-oscillation control. Two providers configured as mutual fallbacks can bounce between failure surfaces without cooldown and a shared attempt budget.

Idempotency binds the same intent, scope, and operation identity

Idempotency is one foundation of safe retries. It does not mean that a request can be repeated at any future time.

A high-risk operation identity commonly binds:

tenant, actor, and business intent
canonical target and argument digest
operation version
idempotency key
provider and account scope
deduplication-retention window
preconditions and expected receipt

{
  "operation_id": "refund-op-8f31",
  "idempotency_key": "order-4821-refund-v1",
  "request_hash": "sha256:illustrative-only",
  "actor": "support-agent-17",
  "tenant_id": "tenant-a",
  "action": "create_refund",
  "target": "payment_9027",
  "amount": 12800,
  "currency": "TWD",
  "precondition_version": 18,
  "status": "intent_committed",
  "provider_request_id": null,
  "dedupe_valid_until": "2026-07-09T10:00:00Z",
  "expected_receipt": "refund_id"
}

The example shows that intent, canonical request hash, and the deduplication boundary should be persisted before the side effect. It does not establish provider support, correct clocks or retention, or continuing validity of the business preconditions.

Stripe stores the first result for an idempotency key and rejects reuse with different parameters. Its documentation also gives the key a retention period, after which reuse may create a new request. That is a provider contract, not a permanent cross-system guarantee.²³

“Always reuse the same key for the same business intent” is therefore incomplete. Safety requires that:

canonical intent has not changed
the request hash matches
the key remains inside the provider’s deduplication window
actor, tenant, and account scope match
preconditions remain valid
policy still permits retry

If one of these changes, the system needs a new operation version rather than disguising a new intent as an old retry.

Idempotent does not necessarily mean retry-safe. The eventual API state may be idempotent while approval has expired, inventory has changed, credentials have been revoked, or the deadline has passed. Retry safety is the wider judgement.

Unknown outcome: inspect authority before deciding whether to resend

Unknown outcomes are easily amplified by “try again”.

Unknown outcome reconciliation flow

Figure 8-2 | After a timeout, the harness persists an unknown outcome and consults external authority under the provider contract. not_found supports retry only when consistency and deduplication conditions make that conclusion valid.

Intent committed
→ request submitted
→ external system may execute
→ response lost or deadline exceeded
→ local state = unknown

The harness reconciles through an operation identifier, idempotency key, provider request identifier, business key, or status API. The result may be:

completed
not_started_confirmed
not_found_inconclusive
partial
failed
unknown

The dangerous simplification is treating not_found as not_started. A status index may be eventually consistent, limited to one region, or incomplete for the relevant time window. Whether negative evidence permits retry is part of the provider contract.

A safe next transition is usually:

completed: store the receipt and continue the local state transition.
not_started_confirmed: retry only while budget, preconditions, and approval remain valid.
not_found_inconclusive: wait for the consistency window, use another lookup, or retain unknown.
partial: stop later irreversible work and enter forward repair, compensation, or human review.
unknown: keep the state unknown and claim neither success nor failure.

If a provider explicitly guarantees deduplication for the same idempotency key inside a valid window, the system may resend the identical canonical request while the outcome remains unknown. That is a contract-backed deduplicated resubmission, not a blind retry. Without that guarantee, reconcile or escalate first.

Stripe’s low-level error guidance similarly notes that network faults can occur at several points in the request lifecycle, requiring idempotency and later state inspection.³

Exactly-once is normally a local property

Within one database transaction, constraints and atomic commit can control writes. Once a path crosses a network, queue, worker, and third-party API, no single component usually guarantees end-to-end exactly-once execution.

A practical design combines:

at-least-once delivery
stable operation or occurrence identifiers
idempotent consumers
inbox, outbox, or durable operation records
deduplication windows
reconciliation
compensation or forward repair
audit receipts

A queue may redeliver. A worker may complete an external operation and crash before acknowledging the message. A scheduler may repeat a trigger after failover. Every boundary must decide whether it represents the same intent rather than trusting “this is the first time I saw the message”.

Accordingly, exactly-once should be scoped to an explicit boundary, such as “one database unique key commits once”, rather than presented as a guarantee spanning the entire system.

Compensation and sagas: corrective actions can fail too

An email may already be delivered, a refund may have entered settlement, a deployment may have served traffic, or a webhook may have reached a third party. A database rollback cannot return the whole world to the same state.

The system may choose:

forward repair
a reversal transaction
snapshot restoration
disablement or quarantine
corrective notification
manual reconciliation

Saga compensation chain

Figure 8-3 | Each forward step and compensation retains its own receipt, idempotency, and failure state. Recovery does not guarantee a return to the original world and need not always run in reverse order.

A saga divides a long transaction into local transactions. Each step should define:

forward action and precondition
durable receipt
retry owner
compensation or forward-repair action
compensation idempotency key
compensation receipt and failure state
dependency order
point of no return
manual escalation path

Compensation is a new side effect and can time out, become partial, or remain unknown. It needs its own recovery state machine rather than an audit field saying rolled_back: true.

Reverse order is common, not universal. An email normally cannot be “unsent”. If a refund completed and the local order update failed, forward repair of local state is more appropriate than another financial reversal.

Create refund
→ update order state
→ send customer email

Refund succeeds and order update fails: retain the refund receipt and repair local state.
Email fails: retry the notification; do not repeat the refund.
Refund is partial: freeze later notification and enter compensation or manual review.

The value of a saga is not automatic time travel. It turns “the workflow failed” into local states that can be judged, traced, and recovered independently.

A circuit breaker protects a dependency; it does not choose every retry

When a provider fails continuously, retries increase queueing, latency, and cost. A circuit breaker fails new calls quickly after error rates or other health signals cross a threshold.

Common states are:

closed: calls pass normally
open: new calls fail fast for a period
half_open: a small number of probes test recovery

Retry and circuit breaker patterns have different responsibilities: retries attempt to cross a transient fault, while a breaker prevents repeated calls that are likely to fail. Microsoft’s official pattern makes the same distinction.⁴

A reliable implementation needs:

an explicit failure taxonomy that excludes business denials and validation failures from dependency-health signals
a sensible scope by provider, operation class, region, or tenant
rolling windows, cooldown, and probe budget
isolated capacity for half-open probes
shared state or explicit instance-local semantics
fallback, queue, rejection, or degraded paths
recovery evidence

A scope that is too broad lets one endpoint open the circuit for an entire provider. A scope that is too narrow lets every worker rediscover the same outage. Breaker state, retry budget, and admission control need coordination; otherwise one control says “stop” while another continues scheduling background retries.

Admission control: some work should not enter

Agent workloads can occupy models, sandboxes, browsers, external APIs and workers for minutes. A conventional request-per-second limit does not capture a run that lasts ten minutes, creates ten subagents and issues hundreds of tool calls.

Before admission, a harness may evaluate:

tenant quota
global and per-tenant concurrency
model and tool capacity
sandbox availability
estimated cost
priority
deadline
required region or credential
side-effect risk

Accepting work when capacity is already unavailable only converts an immediate rejection into a much later failure.

Backpressure, bulkheads and load shedding

Admission control, backpressure and bulkheads

Figure 8-4 | Admission control protects the entrance, backpressure slows producers, bulkheads isolate resource pools, and load shedding preserves control and critical work during overload.

Backpressure tells the producer that the consumer is full

When a downstream service slows, the harness can:

use bounded queues
reduce parallelism
pause new subagents
delay low-priority work
reject tasks that can no longer meet their deadline
return queue position or Retry-After
move to a compatible provider with available capacity

Google SRE treats overload as an explicit reliability problem and recommends early rejection, load shedding and degraded responses where appropriate. Allowing work to accumulate without a bound causes latency, timeouts and retries to reinforce one another.⁵

Bulkheads contain the blast radius

Resource pools can be separated by:

tenant
interactive versus batch work
read-only versus side-effect work
high-cost versus lower-cost models
external provider
evaluation traffic
region

A large batch job for one tenant should not consume the capacity needed for interactive approvals and cancellation.

Load shedding should preserve control paths

Overload priority should not be based only on queue arrival time. A sensible order often preserves:

kill switches and cancellation
approval responses
reconciliation for high-risk side effects already in flight
active critical work
interactive requests
batch and speculative work

A particularly bad overload design puts cancellation and incident inspection behind the same saturated queue as the work causing the overload. The system loses control, and the brakes are also waiting for a slot.

Async work separates acceptance, execution, retention, and delivery

Slow tools, indexing jobs, benchmarks, background agents, and large extraction tasks need durable work identity.

Common execution states include:

accepted → queued → running → completed
                            ↘ failed
                            ↘ cancelled
                            ↘ expired
                            ↘ unknown

A work unit should retain:

work identifier, type, and parent run
owner and claim generation
created, queued, started, heartbeat, and terminal timestamps
current state and state version
output or error reference
retry, idempotency, and cancellation metadata
notification and delivery state
retention policy

Background acceptance is the terminal result of the original tool call

{
  "work_id": "work-391",
  "status": "accepted",
  "inspection_uri": "artifact://work-391/status",
  "cancellable": true,
  "side_effect_may_have_started": false
}

A later completed, failed, or cancelled record is a work-lifecycle event, not a second result for the original tool request.

Terminal work state is also distinct from result delivery. Separate at least:

work terminal state committed
result artefact retained
completion event written to an outbox or event log
subscriber delivery attempted
subscriber acknowledgement or parent acceptance where the protocol supports it
retention or tombstone transition

Result retention cannot depend on a vague “notified” flag

Some protocols have no reliable end-to-end acknowledgement. The system must not pretend that it knows the user saw the result. It should retain delivery records, attempts, subscriber cursors, and result expiry, allowing pull and replay.

Cleanup can proceed in stages:

release large sandbox, browser, and worker resources
retain terminal state and result artefact
publish completion through a durable outbox
reduce the record to an audit tombstone according to acknowledgement, cursor, and retention policy

notification_sent proves at most that a notification service accepted a message. It does not prove that the parent processed it, the user saw it, or the result was accepted.

Durable execution replays workflow state, not external side effects

Durable execution restores a workflow from event history and committed progress after process, worker, or infrastructure failure. Temporal’s model centres on durable workflow execution, event history, and deterministic replay.⁶

The boundary remains important:

workflow orchestration code requires determinism and versioning
external APIs, databases, and other non-deterministic work belong at activity or tool boundaries
an activity may still execute at least once
side effects still need idempotency and unknown-outcome reconciliation
workflow upgrades require versioning or patching
invalid business input does not become valid through durable retry

Temporal’s official versioning guidance similarly states that replay requires deterministic workflow code and that long-running executions need patching or Worker Versioning.⁷

Durability preserves “the workflow knows where it reached”. It does not automatically provide external exactly-once execution, compensation, or correct business decisions.

Layered context degradation: context limits need a recovery order

Context failure is also a reliability problem.

When model context approaches its limit, the harness should not send the entire transcript into one opaque summarisation pass. A safer degradation order is:

persist oversized raw artefacts
replace old reproducible tool payloads
remove low-value middle dialogue
create a schema-bound semantic summary
use emergency compaction if necessary
stop safely when critical state can no longer be preserved

Cutting must not separate:

tool requests from correlated results
approval requests from decisions
background acceptance from work IDs
side-effect operations from receipts
task claims from ownership records

A summary should retain at least:

current objective
user constraints
decisions and reasons
modified resources
tests and evidence
open failures
active operation IDs
remaining work
next safe action

If the result remains too large or compaction keeps failing, the harness needs a circuit breaker. Repeatedly asking a model to summarise the summary of its previous summary tends to produce thinner evidence rather than concentrated truth.

Scheduled work gives every occurrence its own identity

A scheduled agent task should not run the whole workflow inside a timer callback.

Schedule definition
→ occurrence created
→ queue delivery
→ agent consumer executes
→ terminal result retained
→ completion delivered
→ subscriber notified

Async and scheduled delivery lifecycle

Figure 8-5 | Schedule definition, occurrence, queue delivery, work execution, result retention, and notification are distinct states. Redelivery does not create a new business intent.

A job definition should include:

job identifier, schedule, and time zone
tenant, owner, and payload reference
recurring or one-shot mode
misfire and catch-up policy
overlap and concurrency policy
expiry and deadline
allowed agent-configuration version
approval requirement
schedule version

Each trigger also needs:

occurrence identifier
scheduled-for timestamp
deduplication or idempotency key
definition version
delivery attempt
work identifier

Queue redelivery and scheduler failover can then recognise “the same occurrence delivered again” instead of producing two reports or two payments.

Durable definition is not durable execution

Durable definition: the schedule survives restart.
Durable trigger: a due occurrence is persisted by an external scheduler or store.
Durable execution: worker failure resumes from committed progress.
Durable delivery: a terminal result can be pulled or replayed after notification failure.

A stored cron expression proves only that the system remembers when it intended to work.

Misfire, catch-up, and overlap require explicit policy

After a missed time, the system may:

skip
fire once now
replay missed occurrences
coalesce into one run

When a previous occurrence remains active, it also chooses whether to:

skip the new occurrence
buffer one or all
cancel the previous occurrence
allow overlap

Temporal’s schedule documentation likewise treats overlap policy and catch-up windows as separate controls; poor choices can create missed runs or a burst of replayed work after recovery.⁸

Time zones and daylight-saving changes also belong in the runtime contract. 02:30 may not exist or may occur twice on some dates, so occurrence identity should retain the actual instant and original schedule semantics.

Failure testing verifies invariants, not merely eventual success

Reliability tests should inject faults at precise boundaries:

before and after intent commit
before and after request submission
after an external side effect but before receipt persistence
after work completion but before queue acknowledgement
during compensation
during a half-open circuit probe
around schedule-trigger persistence
around notification-outbox commit

Cover at least:

model and tool timeouts
rate limits and provider outages
duplicate delivery
partial batch failure
worker crash and checkpoint recovery
unknown outcome and eventually consistent status queries
stale-claim submission
queue backlog and retry storms
open and half-open circuits
context-compaction failure
scheduled misfire and overlap
notification redelivery

The test should inspect more than whether the workflow eventually succeeded:

duplicate side effects
bounded attempts, cost, and deadline
recoverable run state
complete receipts or explicit unknown states
queue amplification from retries
honest user-visible state
effective stop and quarantine paths
compensation audit trails

Chaos testing is not complete because several workers were killed at random. Each test needs a hypothesis, injected fault, expected transition, forbidden outcome, and recovery evidence. Without an invariant, fault injection is only noise.

Walking through the refund recovery

Return to the refund at the beginning.

1. Persist the intent

The harness creates an operation_id and idempotency key, then stores the order, amount, tenant, actor, policy decision and expected receipt.

2. Send the refund request

The tool adapter calls the payment provider. Run state becomes submitted, not completed.

3. The call times out

The classifier records UNKNOWN_SIDE_EFFECT_OUTCOME, not an ordinary TOOL_TIMEOUT.

4. Reconcile

The harness queries refund status using the idempotency key, provider request ID or payment ID.

If completed, store the refund receipt and update local order state.
If confirmed not started, retry within budget using the same idempotency key.
If partial, pause the customer email and start compensation or human review.
If still unknown, retain unknown, poll later and do not claim completion.

5. The order-state update conflicts

The database reports a version conflict. This is not a reason to issue another refund. The harness reloads canonical state and retries only the local state transition.

6. Customer notification fails

The email provider rate-limits the notification. The email operation uses its own message ID and follows Retry-After. The refund is not repeated.

7. Completion gate

The verifier checks:

the provider refund receipt
canonical order refund state
notification state
operation record and audit trail

The model’s sentence “the refund has been processed” is not completion evidence.

This single run encountered an unknown outcome, a state conflict and a rate limit. All three were errors. None required the same recovery action.

Fifteen questions for reviewing a reliability harness

Does failure state include both domain and side-effect certainty?
Which component owns recovery decisions?
Do logical attempts, SDK retries, and queue redelivery enter one ledger?
Is the deadline propagated downwards, or restarted at every layer?
Which operations define idempotency scope, request hash, and deduplication window?
Does retry safety re-check approval, preconditions, credentials, and budget?
How is external authority queried after a timeout, and can not_found be merely not-yet-consistent?
Can an unknown outcome remain unknown rather than being guessed into success or failure?
Does compensation have its own idempotency, receipt, failure state, and manual path?
Does fallback preserve tools, policy, region, evidence quality, and the completion contract?
Do circuit breaker, retry budget, and admission control use a coherent scope?
Is capacity reserved for cancellation, approval, and reconciliation?
Are background completion, result retention, delivery, and parent acceptance separate?
Does each scheduled occurrence have stable identity, overlap, misfire, catch-up, and expiry policy?
Have duplicate delivery, crash, unknown, partial, and overload faults been tested at commit boundaries?

If the main answers remain “run it again”, “the queue handles that”, “the workflow engine probably remembers”, or “check the result at the end”, reliability is still based on hope.

From theory to implementation: Part 08 checklist

Part 09 addresses a separate boundary: an operation may execute reliably and still lack authority. The next article builds the safety chain from identity, permission, approval, sandboxing, credentials, tenant boundaries, and egress policy.

References

Amazon Web Services, Timeouts, retries, and backoff with jitter, accessed 8 July 2026. ↩ ↩²
Stripe, Idempotent requests, accessed 8 July 2026. ↩
Stripe, Advanced error handling, accessed 8 July 2026. ↩ ↩²
Microsoft Azure Architecture Center, Circuit Breaker pattern, accessed 8 July 2026. ↩
Google, Handling overload and Addressing cascading failures, accessed 8 July 2026. ↩
Temporal, Temporal Platform Documentation and Understanding Temporal, accessed 8 July 2026. ↩
Temporal, Safely deploying changes to Workflow code, accessed 8 July 2026. ↩
Temporal, Schedule and Troubleshoot missed Schedule Actions, accessed 8 July 2026. ↩