Evolvex Remote Operations

Correctness infrastructure for durable execution, ambiguity detection, safe recovery, and reconciliation of irreversible remote operations in Laravel.

Use it when a remote side effect may have executed even though your application did not receive a response: withdrawals, refunds, provider wallet operations, casino actions, affiliate payouts, SMS sends, webhooks, external provisioning, and similar operations.

This package does not promise exactly-once execution against arbitrary external systems. It prevents blind retries, makes ambiguous outcomes explicit, uses provider idempotency/status probes when available, and escalates cases that cannot be proven safe.

Guarantees and invariants

• UNKNOWN is a first-class state, never treated as a normal exception.
• A business operation is locally unique by (type, business_key).
• A retry cannot silently mutate the original command payload.
• The same provider idempotency key cannot be bound to two different operations; a DB unique constraint closes the concurrent race window.
• Remote side effects and direct queue handoffs are blocked while a DB transaction is open.
• Durable async dispatch and all automatic retries are persisted through an outbox.
• Execute and reconcile jobs are generation-fenced; stale delayed jobs cannot consume a future retry.
• Outbox queue handoffs have a reclaim lease, so a lost execute job is safely redelivered instead of stranding a READY operation.
• A crashed send worker is recovered from EXECUTING to UNKNOWN after its lease expires.
• A crashed probe worker is recovered from RECONCILING and safely probed again.
• Unknown outcomes are probed before retry whenever the provider supports a status probe.
• NOT_FOUND can remain non-definitive for eventually-consistent providers.
• Idempotent retry respects the provider's idempotency TTL measured from the first send attempt.
• Provider safety capabilities are snapshotted at first send so a later deployment cannot silently reinterpret an in-flight ambiguous operation.
• Capability contracts are validated before the first send; declaring status-probe support without implementing ProbesRemoteOperations fails before any remote side effect.
• Probe limits/backoff are scoped to the current send episode; a new send gets a fresh ambiguity/probe window.
• next_action_at is enforced by workers, so stale/early jobs cannot bypass configured backoff.
• Manual/unsafe retry requires explicit duplicate-risk acknowledgement and is audited.
• State transitions are stored append-only until the configured whole-operation retention window expires.
• Terminal command payload/idempotency plaintext can be scrubbed earlier while fingerprints/hashes remain for duplicate detection.
• Events implement Laravel's ShouldDispatchAfterCommit contract.
• Sensitive command payloads and idempotency keys are encrypted at rest using Laravel's app key.

Installation

composer require evolvex/laravel-remote-operations
php artisan migrate

Optional publishing:

php artisan vendor:publish --tag=remote-operations-config
php artisan vendor:publish --tag=remote-operations-migrations

Requirements: PHP 8.3+ and Laravel 12/13.

Define a durable provider handler

Do not capture an in-memory closure as the only way to contact a provider. Reconciliation may happen in another worker hours later.

<?php

namespace App\RemoteOperations;

use Evolvex\RemoteOperations\Contracts\ProbesRemoteOperations;
use Evolvex\RemoteOperations\Contracts\RemoteOperationHandler;
use Evolvex\RemoteOperations\Domain\Enums\RetrySafety;
use Evolvex\RemoteOperations\Domain\Enums\TransportPhase;
use Evolvex\RemoteOperations\Domain\ValueObjects\ProbeConsistency;
use Evolvex\RemoteOperations\Domain\ValueObjects\ProbeOutcome;
use Evolvex\RemoteOperations\Domain\ValueObjects\ProviderCapabilities;
use Evolvex\RemoteOperations\Domain\ValueObjects\RemoteOperationContext;
use Evolvex\RemoteOperations\Domain\ValueObjects\SendOutcome;
use Illuminate\Http\Client\ConnectionException;
use Illuminate\Support\Facades\Http;

final class ProviderXWithdrawal implements RemoteOperationHandler, ProbesRemoteOperations
{
    public function provider(RemoteOperationContext $context): string
    {
        return 'provider-x';
    }

    public function capabilities(RemoteOperationContext $context): ProviderCapabilities
    {
        return ProviderCapabilities::make(
            supportsIdempotency: true,
            supportsStatusProbe: true,
            retrySafety: RetrySafety::IDEMPOTENT,
            idempotencyTtlSeconds: 86_400,
            probeConsistency: ProbeConsistency::eventual(
                visibilityDelaySeconds: 5,
                definitiveNotFoundAfterSeconds: 120,
            ),
            maxSendDurationSeconds: 30,
        );
    }

    public function send(RemoteOperationContext $context): SendOutcome
    {
        try {
            $response = Http::withHeaders([
                'Idempotency-Key' => $context->idempotencyKey(),
            ])->timeout(30)->post('https://provider.example/withdrawals', $context->command);
        } catch (ConnectionException $e) {
            return SendOutcome::unknown(
                reason: 'connection_exception',
                phase: TransportPhase::UNKNOWN,
                metadata: ['endpoint' => 'https://provider.example/withdrawals', 'http_method' => 'POST'],
            );
        }

        if ($response->successful() && $response->status() !== 202) {
            return SendOutcome::success(
                externalId: $response->json('id'),
                metadata: [
                    'provider_status' => $response->json('status'),
                    'provider_request_id' => $response->header('X-Request-Id'),
                    'endpoint' => 'https://provider.example/withdrawals',
                    'http_method' => 'POST',
                ],
                httpStatus: $response->status(),
            );
        }

        if ($response->status() === 202) {
            return SendOutcome::remotePending(
                externalId: $response->json('id'),
                metadata: ['provider_status' => 'processing'],
            );
        }

        if ($response->status() === 422) {
            return SendOutcome::definiteFailure('provider_validation_rejected', httpStatus: 422);
        }

        return SendOutcome::unknown(
            reason: 'provider_5xx_with_unknown_commit_state',
            phase: TransportPhase::RESPONSE_RECEIVED,
            httpStatus: $response->status(),
        );
    }

    public function probe(RemoteOperationContext $context): ProbeOutcome
    {
        $response = Http::get('https://provider.example/withdrawals/by-reference/'.$context->businessKey());

        if ($response->status() === 404) {
            return ProbeOutcome::notFound();
        }

        return match ($response->json('status')) {
            'completed' => ProbeOutcome::succeeded($response->json('id')),
            'failed' => ProbeOutcome::failed('provider_failed'),
            default => ProbeOutcome::pending($response->json('id')),
        };
    }
}

Synchronous execution

use Evolvex\RemoteOperations\Facades\RemoteOperation;

$result = RemoteOperation::for('withdrawal', $withdrawal->uuid)
    ->handler(\App\RemoteOperations\ProviderXWithdrawal::class)
    ->subject($withdrawal)
    ->idempotencyKey($withdrawal->uuid)
    ->externalReference('withdrawal:'.$withdrawal->uuid)
    ->command([
        'withdrawal_id' => $withdrawal->id,
        'reference' => $withdrawal->uuid,
        'amount' => (string) $withdrawal->amount,
        'currency' => $withdrawal->currency,
    ])
    ->execute();

execute() is intentionally rejected when any resolved Laravel DB connection has an open transaction.

Durable asynchronous execution

dispatch() uses the outbox by default:

$operation = RemoteOperation::for('refund', $refund->uuid)
    ->handler(ProviderXRefund::class)
    ->idempotencyKey($refund->uuid)
    ->command([...])
    ->dispatch();

If you explicitly want a non-outbox queue handoff, use dispatchDirect(). It is blocked inside DB transactions.

Transactional outbox

DB::transaction(function () use ($withdrawal) {
    RemoteOperation::for('withdrawal', $withdrawal->uuid)
        ->handler(ProviderXWithdrawal::class)
        ->idempotencyKey($withdrawal->uuid)
        ->command([...])
        ->dispatchAfterCommit();
});

By default, if the active business transaction is on a different database connection from the remote-operation tables, the package throws NonAtomicOutboxException.

State model

CREATED -> READY -> EXECUTING
                     |  |  |
                     |  |  +-> CONFIRMED_FAILURE
                     |  +----> REMOTE_PENDING -> RECONCILING
                     +-------> UNKNOWN --------> RECONCILING

RECONCILING -> CONFIRMED_SUCCESS
            -> CONFIRMED_FAILURE
            -> REMOTE_PENDING
            -> UNKNOWN
            -> RETRY_SCHEDULED -> READY
            -> MANUAL_REVIEW

Recovery rules

Scheduler

use Illuminate\Support\Facades\Schedule;

Schedule::command('remote-operations:dispatch-outbox')->everyMinute()->onOneServer();
Schedule::command('remote-operations:recover-stale')->everyMinute()->onOneServer();
Schedule::command('remote-operations:reconcile-due')->everyMinute()->onOneServer();
Schedule::command('remote-operations:prune')->daily()->onOneServer();

Operational commands

php artisan remote-operations:list --state=unknown
php artisan remote-operations:show 01K...
php artisan remote-operations:reconcile 01K...
php artisan remote-operations:reconcile-due
php artisan remote-operations:recover-stale
php artisan remote-operations:dispatch-outbox
php artisan remote-operations:stats
php artisan remote-operations:doctor
php artisan remote-operations:providers
php artisan remote-operations:prune

Manual resolution:

php artisan remote-operations:resolve 01K... success \
  --external-id=TX-123 \
  --reason="confirmed by provider support" \
  --actor=admin:42

Unsafe retry:

php artisan remote-operations:retry 01K... \
  --acknowledge-duplicate-risk \
  --reason="provider confirmed operation never existed" \
  --actor=admin:42

Security

• Command payload is stored using Laravel's encrypted cast.
• Idempotency key is encrypted; a SHA-256 hash is stored separately.
• Attempt/probe metadata passes through MetadataRedactor.
• Raw endpoint metadata is hashed for attempt correlation and removed from stored metadata.
• Do not place PAN/CVV/secrets in metadata.

Testing

composer lint
composer test

Limits

• This is not a payment gateway SDK.
• This is not a generic Saga/workflow engine.
• The provider adapter must classify evidence conservatively.
• A provider with neither idempotency nor a reliable status/reference lookup cannot be made exactly-once. MANUAL_REVIEW is the safe result.

See docs/correctness-model.md, docs/provider-contract.md, docs/failure-matrix.md, and docs/operations-runbook.md.