UCP Agent Failure Recovery: Building Resilient Commerce Systems Without Cascading Failures

The posts on your site cover UCP error handling, retry logic, observability, and webhook security—but none address the complete failure recovery architecture that separates production-grade commerce agents from systems that collapse under real-world conditions.

When a commerce agent encounters a failure—a payment gateway timeout, an inventory service outage, or a malformed response from a fulfillment partner—the stakes are not abstract. A transaction in progress can be lost. A customer order can be duplicated. A payment can be charged twice. Recovery requires more than logging and retries.

The Failure Recovery Gap in UCP Architectures

Most UCP integrations treat failure as a binary: succeed or fail and retry. But commerce operates on state. An agent may have successfully charged a payment but failed to create an order in the fulfillment system. A second retry may create a duplicate order. A third attempt may fail because the payment was already processed. Without explicit recovery state tracking, each retry becomes a gamble.

This gap exists because failure recovery in UCP requires:

• Transaction idempotency tracking: Recording which UCP operations succeeded, failed partially, or succeeded with side effects.
• State machines per transaction: Knowing whether a payment is committed, an order is created, an inventory hold is active, or a shipment is pending.
• Compensation logic: Reversing side effects when downstream systems fail (e.g., releasing inventory holds, refunding charges, canceling orders).
• Agent memory across retries: Ensuring the agent remembers what it attempted and why, not re-attempting the same failed operation.
• Circuit breakers and fallback routing: Knowing when to stop retrying a system and route to an alternative provider or manual review.

Designing UCP Transaction State Machines

Every UCP commerce agent interaction should map to a finite state machine (FSM) that tracks progress through discrete stages. A typical order transaction has these states:

1. Payment Authorization (not yet committed, can be abandoned)
2. Payment Captured (committed, requires refund to reverse)
3. Order Created in Fulfillment System (requires cancellation to reverse)
4. Inventory Reserved (requires release to reverse)
5. Shipment Initiated (requires cancellation request)
6. Delivered or Failed (terminal state)

Each state transition must be recorded in a durable log (not just in agent memory). If the agent crashes or is re-invoked after a network failure, it reads this log and knows exactly where it left off.

// Pseudocode: UCP Agent Transaction Log
{
  "transaction_id": "txn_abc123",
  "timestamp": "2026-03-12T14:32:00Z",
  "states": [
    {
      "stage": "payment_authorization",
      "status": "success",
      "provider": "stripe",
      "provider_id": "pi_1234",
      "timestamp": "2026-03-12T14:32:05Z"
    },
    {
      "stage": "payment_capture",
      "status": "success",
      "provider": "stripe",
      "amount": 9999,
      "timestamp": "2026-03-12T14:32:10Z"
    },
    {
      "stage": "order_creation",
      "status": "failed",
      "provider": "shopify",
      "error": "timeout",
      "retry_count": 2,
      "last_attempted": "2026-03-12T14:32:45Z"
    }
  ]
}

When the agent detects that payment capture succeeded but order creation failed, it has two options: retry order creation (idempotent, using the same order ID), or trigger compensation logic (refund the payment, mark transaction as failed).

Idempotency Keys and Duplicate Prevention

UCP agents must use idempotency keys for every write operation. An idempotency key is a unique identifier that the agent provides to external systems (payment gateways, fulfillment platforms, inventory services). If the network fails and the agent retries the same operation, the external system recognizes the idempotency key and returns the cached result instead of processing the request twice.

Payment example: An agent calls Stripe with idempotency_key = “order_abc123_payment_capture”. Stripe receives the request, captures the payment, and stores the result. The network fails. The agent retries with the same idempotency_key. Stripe recognizes the key, returns the cached result, and does not charge the customer again.

Idempotency must extend beyond payment gateways. Shopify’s API accepts idempotency keys for order creation. Custom fulfillment APIs should implement the same pattern. Without it, retries inevitably produce duplicates.

Compensation and Rollback Strategies

When a transaction progresses partway and then fails irreversibly, the agent must reverse completed steps. This is the Saga pattern in distributed systems.

Example failure chain:

1. Agent charges payment successfully (captured).
2. Agent attempts to create order in fulfillment system; system is down for 30 minutes.
3. Agent exhausts retry budget and implements compensation: refund the payment.
4. After refund completes, the agent either notifies the customer that the order failed or routes to manual review.

Compensation logic is not automatic—it requires explicit coding. Common compensation patterns:

• Payment refund: Call payment.refund(charge_id) if order creation fails.
• Inventory release: Call inventory.release_hold(hold_id) if payment fails.
• Order cancellation: Call fulfillment.cancel_order(order_id) if shipment initiation fails.
• Manual escalation: If compensation itself fails, log the transaction and alert operations.

Circuit Breakers and Fallback Routing

An agent should not retry indefinitely against a failing system. Circuit breaker logic stops retries after a threshold is reached and routes traffic to an alternative provider or manual review.

Circuit breaker rules for payment gateways:

• If Stripe returns 5 consecutive authorization failures → open circuit, route to Adyen.
• If Stripe times out on 10 consecutive requests → open circuit for 5 minutes, then try again.
• If Stripe reports “insufficient funds” → do not retry, escalate to customer support.

Without circuit breakers, a single failing provider can paralyze an entire commerce system as agents waste cycles retrying.

Agent Memory and Retry Context

When a UCP agent is re-invoked (either by the framework after a crash or by the merchant after manual recovery), it must have access to the full context of prior attempts. This includes:

• What operations were attempted and their outcomes.
• How many times each operation has been retried.
• Which fallback providers have been exhausted.
• The exact error messages and error codes from failed systems.

This context should be stored in the transaction log and made available to the agent via the UCP API. Without it, the agent has no way to make intelligent retry decisions—it may retry the same failed operation against the same failing provider.

FAQ: Common Failure Recovery Questions

1. How do I know if a payment was actually charged if the response times out?

You query the payment gateway’s API with the idempotency key. If the charge was created, the gateway returns the charge ID and status. If it was not created, the gateway returns null. Always implement this fallback query before triggering compensation.

2. What if compensation itself fails (e.g., refund request times out)?

This is a critical edge case. Log the transaction in a “compensation_failed” state and alert operations immediately. Manual intervention is required. Do not retry compensation automatically, as this can lead to double-refunds.

3. Should I retry every type of error?

No. Retryable errors (timeouts, network failures, 5xx responses) warrant retry. Non-retryable errors (invalid customer data, 4xx responses, business rule violations) should trigger compensation or manual review immediately.

4. How long should I keep transaction logs?

At minimum, 7 days (to cover chargeback windows). Ideally, 90 days (to support audits and customer service inquiries). Store logs in a durable database, not in agent memory.

5. How do I test failure recovery without taking production down?

Implement chaos engineering tests: disable Stripe endpoints and verify refund logic works; disable inventory service and verify order cancellation works. Run these tests in staging with production-like data volumes.

6. What if two agents try to recover the same transaction simultaneously?

Use distributed locks (Redis, DynamoDB conditional writes) to ensure only one agent attempts recovery at a time. Lock the transaction ID before reading state and updating it.

7. How do I prevent the customer from resubmitting their order while recovery is in progress?

Return a 202 Accepted response immediately, with a status URL the customer can poll. Once recovery completes (success or failure), update the status URL. This prevents duplicate submissions.

Conclusion: Resilience as a Design Requirement

UCP agents operating in production must treat failure recovery as a first-class design concern, not an afterthought. This requires transaction state tracking, idempotency keys, compensation logic, circuit breakers, and durable logs. The merchant integrations that will scale are those that architect for failure from day one, not those that discover failure recovery requirements during their first production incident.