[Bug]: manifest_worker fails closed on transient facilitator settle timeouts instead of flagging payment-orphaned

[Meenbudha]

Describe the bug clearly

During periods of network congestion on EVM chains (e.g., Base mainnet/Sepolia), block confirmation times can exceed the facilitator client's internal timeout (typically 5–10 seconds). When the facilitator's /settle endpoint times out or fails during transmission, the _settle_payment method in manifest_worker.py logs a settlement failure and returns metadata marking the task payment status as payment-failed.

However, since the signed EIP-3009 authorization payload was already broadcasted, the transaction can still successfully confirm on-chain shortly after the timeout. When this happens:

1. The client's wallet is debited.
2. The agent's wallet is credited.
3. The agent's task record is permanently stuck as failed (with status payment-failed rather than payment-orphaned), meaning the user paid for a service that will never execute, and no alert is triggered for the operator.

To Reproduce

1. Configure a Bindu agent with x402 payments enabled on a live network (e.g., Base Sepolia).
2. Initiate a task request to the agent.
3. Simulating network congestion, introduce a mock delay or force the HTTP facilitator client to return a timeout exception during /settle.
4. Observe the console log registering the settlement error and transitioning the task state to failed with payment-failed metadata.
5. Check the target EVM block explorer or the agent's wallet to observe that the transaction succeeded on-chain, indicating a successful credit transfer.

Expected behavior

• The system should distinguish between a facilitator network timeout (transient) and a hard API rejection (permanent).
• If a facilitator /settle call fails or times out, the task should eventually be picked up by a background reconciliation worker. This worker should check the blockchain for the AuthorizationUsed event matching the saved EIP-3009 nonce, updating the status to payment-orphaned or triggering an automated/manual refund.

Screenshots or logs

[21:30:40] INFO     Starting deployment for agent: 438b4815-7ebe-d853-b95d-48b32b68fa3a
...
[21:30:52] ERROR    Error settling payment: HTTPFacilitatorClient.settle timed out after 10.0s
[21:30:52] INFO     Payment settlement failed; task not executed.

Environment

• OS: Windows 10 / POSIX
• Python version: Python 3.12+ (tested on Python 3.14.0)
• Package version: bindu core v0.3.15.dev1519

Additional context
The root of the issue is in bindu/server/workers/manifest_worker.py (inside the _settle_payment and _handle_settlement_failure methods). When _settle_payment encounters an exception (such as a timeout), it falls back to:

        except Exception as e:
            logger.opt(exception=True).error("Error settling payment: {}", e)
            return _failure_metadata(str(e))

This returns _failure_metadata, which tags the status as payment-failed instead of payment-orphaned. The worker has no secondary mechanism to check if the transaction is pending or eventually confirmed.

---

Suggested Fixes & Improvements

1. Reconciliation Loop (Detection):
   Implement a background reconciliation worker (similar to ManifestWorker but scheduled to run every few minutes). This worker should:

   • Query the task storage for tasks marked payment-failed within the last 24 hours.
   • Parse the stored EIP-3009 metadata (x402_nonce, x402_authorization.from, and x402_network).
   • Perform an eth_getLogs RPC query on the target USDC contract to search for the corresponding AuthorizationUsed event.
   • If the transaction was completed, transition the task's payment status to payment-orphaned to signal the operator.

2. Facilitator Polling / Increased Timeout:
   Introduce a configurable retry or validation step in _settle_payment for timeouts. Before classifying the payment as failed, the client could poll the chain's state to confirm if the signature/nonce was already consumed on-chain.

[Meenbudha]

/assign under ELUSOC
I will like to work on this