S3 Site Replication

With eEKAS, you can replicate S3 data between sites, mirror it to another S3 instance, and archive it in an external backup. This chapter explains how to choose the right protection method, configure it, monitor its operation, and recover your data.

Choose the protection method

Method	Purpose
eEKAS to eEKAS sync	Asynchronous native site replication: read-only disaster recovery or active-active object writes.
Mirror current objects	One-way current-object copy to another S3 instance. No checkpoint-based restore.
Backup and archive	Retained generations and published recovery checkpoints, with preview and restore.

Site replication propagates changes and does not replace an independent backup. The metadata primary owns administrative metadata; it is not necessarily the only site accepting object writes.

1. Prerequisites and preparation

• Use healthy independent clusters with compatible accepted eEKAS versions. Plan capacity for replicas, retained versions and catch-up.
• Configure each S3 service with local storage, an IP Group, eligible gateway nodes, stable DNS and trusted HTTPS certificates. Verify routing, firewalls, time synchronization and endpoint reachability.
• For native joining, the destination S3 service must be empty, with no buckets. Joining adopts that service; it does not merge populated namespaces.
• For online invitations, enable the source REST API and allow the destination to reach the source management HTTPS listener, normally port 18443.
• Record versioning, Object Lock, retention, legal holds, permissions and SMB gateway configuration. Define recovery authority, fencing, client routing, RPO and RTO.
• For archive jobs, ensure the chosen worker node has source/archive connectivity and enough staging disk.

Keep certificate verification enabled. Configure private CA trust where needed; accepting an untrusted certificate is an explicit exception for controlled private systems.

2. Configure native eEKAS replication

Set up the source installation

1. Open S3 synchronization and recovery → eEKAS to eEKAS sync → Set up source installation.
2. Select the source S3 service and enter a physical site name.
3. Select Disaster recovery (read-only destination), or Active-active (writes allowed at every site) only when applications tolerate concurrent asynchronous writes.
4. Keep peer certificate verification enabled. Choose Enable and wait for the relationship. Active-active requires acknowledgment of write conflicts.

Create a one-time invitation

1. On the source relationship choose Add Site and a one-time online invitation.
2. Set the source management address reachable from the destination, normally HTTPS port 18443.
3. Choose 15 minutes, 60 minutes or 24 hours validity. Create the invitation and transfer its token securely.
4. Use the token before expiry. Review Invitations and revoke unused valid tokens.

The encrypted invitation-file option requires a passphrase of at least 12 characters. Transfer file and passphrase separately. The current destination web form accepts an online address and token; arrange the supported administrative import procedure before choosing the file method.

Join an empty destination

1. At the destination open eEKAS to eEKAS sync → Connect this installation.
2. Select an empty local S3 service and a unique site name. Local storage, IP Group and gateway nodes are used by the replica.
3. Enter the source management address and one-time token. Configure trusted CA information where applicable.
4. Select the intended destination mode, choose Validate and Join, review the endpoint and acknowledge adoption of the empty service.
5. Wait for setup and initial synchronization. Check both installations and read test objects through their actual S3 endpoints. In DR mode, verify destination writes are rejected.

If joining fails, inspect the returned error and Event Log. Partially completed setup may need administrator cleanup; do not delete replica pools or retained original storage to fix a naming error.

3. Policies and daily operation

The evaluated source view lists its local card, while the destination view lists both sites. Card count and the Primary/Secondary labels do not prove remote health. Read the separate synchronization state and gateway count at every site.

Control	Meaning
Recovery point warning threshold	5, 15 or 60 minutes, or 24 hours: a warning objective, not synchronous writes or a guaranteed RPO. Durability RPO remains unmeasured.
Transfer profile	Limited, Balanced or Fast changes concurrency; it is not a hard bandwidth limit.
Synchronization window	Always, or a daily local-time window such as 20:00–06:00. Incoming synchronization pauses outside the window and lag can grow.
Bucket replication scope	All buckets, or Selected buckets separated by semicolons. Unlisted and new buckets are excluded until added.
Advanced Rules	Bucket, key-prefix and tag filters; destination bucket, owner and storage-class mappings; one-way/two-way direction. These replace the simple scope: test routing.
Protection	Review deletion handling, versioning and Object Lock. Retaining previous versions depends on versioning; external copy/deletion-delay settings affect the external transfer path.
Monitoring and Alerts	Email, existing SNMP notifications, webhook state changes or Prometheus metrics. Test notification delivery.
Recovery	Validation, secure administrator-provided recovery hooks and optional global endpoint integration. Manual routing changes still require an administrator.
External Archive	Scheduled current-object copy/mirror. Generation-preserving backup is configured separately under S3 archive and recovery.

Interpret status correctly

Synchronized means reported local incoming streams are caught up. It does not certify the opposite direction, zero data loss or physical-media backup. Check both sites. Syncing/behind requires follow-up, error indicates a problem, and unknown is insufficient evidence.

• Use Refresh and review endpoints, modes, gateways and Event Log.
• Operations offers local per-bucket statistics with Search and Next page, up to 50 entries per page. Counts do not verify the remote inventory.
• Pause/Resume affects local incoming synchronization. In active-active mode, this does not pause all outgoing or remote streams.
• Use Allow Writes or Make Read-only at the relevant secondary; recheck synchronization and applications.

Backfill, credentials and removal

Start Backfill Existing Objects on a destination for objects predating a replication rule. Monitor Backfill Jobs; pause, resume or cancel as needed. Treat backfill as a separate load phase and verify bytes and versions afterward.

Credential rotation: Stage Replacement at the primary → Adopt Replacement at every secondary → Activate Replacement at the primary → Finalize Rotation only after synchronization is healthy. The old key remains valid until finalization.

Removing a secondary from the primary topology is separate from decommissioning its installation. Local decommission stops gateways and removes replica configuration; data is retained unless deletion is explicitly selected and confirmed. Verify recovery alternatives first.

4. Native recovery

Recover S3 gateways

For an unavailable endpoint on a running installation, check gateways, networking and IP Group health. Recover S3 gateways restarts local gateways and refreshes endpoint resources. Review the result, test a real S3 read/write and check catch-up. This does not rebuild lost storage or promote a secondary.

Promote a standby site

1. Identify the intended authority and latest replicated content; stop application writes during the decision where practical.
2. Stop or isolate the old primary and confirm fencing so it cannot resume independent authority.
3. Inspect surviving-site incoming synchronization, gateways and endpoint. Run Validate Recovery Readiness in Operations.
4. Choose Promote or configured Orchestrated Recovery, select the destination, type the displayed zone name and confirm primary isolation.
5. Use emergency force options only after explicitly accepting and recording possible loss of recent objects.
6. Update client routing manually or through tested global endpoint integration. Verify bytes, permissions, WORM and application access.
7. Keep the former primary isolated on return. Refresh topology, reconcile and synchronize before enabling writes or arranging failback.

Recovery hooks and DNS/load-balancer integration need secure executables and tested configuration; a hostname alone does not provide automatic failover. Active-active object writes at a surviving site do not resolve metadata-primary loss. Total-primary-cluster loss and full promotion/failback were not exercised in the recorded evaluation.

5. Mirror current objects to another S3 instance

1. At the metadata-primary installation choose Sync to other S3 instance → Mirror current objects, select the source service and Configure one-way mirror.
2. In External Archive, enable the destination and enter its name, HTTPS endpoint, existing bucket, signing region, access key and secret.
3. Choose Mirror for current content and source deletions, or Copy only to keep extra destination objects. Choose Manual, Hourly or Daily.
4. Review Protection, deletion delay, protected archive, source scope and advanced rules. Save.
5. For a manual run choose Operations → Run Archive Now. Review results and Event Log; verify destination bytes and test overwrite/deletion in dedicated test buckets.

Each source bucket uses a same-named prefix inside the destination bucket. There is no reverse synchronization, checkpoint preview/restore, original-version reconstruction or ownership recovery. Copy only can overwrite an existing destination key. The delayed-deletion area is not a recovery catalogue. Keep unrelated content outside managed prefixes. Qualify listing duration and worker memory for large buckets; new independent external-mirror acceptance remains outstanding.

6. Configure backup and archive

1. Choose Sync to other S3 instance → Configure backup and archive, or New backup under S3 archive and recovery.
2. Enter a unique Backup name, external HTTPS address, existing destination bucket and access/secret keys.
3. Select the source zone. Whole zone covers current and future buckets; Individual buckets opens Available buckets and Selected for backup.
4. Search and add buckets. Selection persists across search/pages; Add this page adds only the displayed page. Remove excludes a selection.
5. Choose Standard backup (recommended), or planned Extra protection for older versions.
6. Choose the eEKAS node under Run this backup on. Create the backup through that node’s management page; verify connectivity and staging space.
7. Review advanced settings, Save backup, then Start backup. Saving alone does not start export. Wait for a completed checkpoint.

Standard backup reads generations still available at the source; it cannot capture a version already removed before reading. Maintain source versioning and WORM. Extra protection creates a local history zone/gateway with separate unused port (for example 9443), HTTPS endpoint, certificate/private key and storage. It adds capacity needs and does not itself survive total primary-site loss.

Advanced setting	Default / use
Source service details and CA	Keep discovered realm, zonegroup, zone and endpoint unless a specific change is needed. Supply the private issuing CA chain where required.
Check for changes every	60 seconds; upload notifications may queue work sooner. No guaranteed export latency.
Minimum time between full recovery catalogs	3,600 seconds. Measure catalogue duration before reducing.
Parallel WORM protection checks	4; metadata checks are separate from uploads. Monitor source request load.
Recheck WORM settings after	3,600 seconds requested minimum; large queues can take longer.
Largest file to stage locally	100 GiB per object; not a total disk reservation. Provide actual workload headroom.
Backup server S3 region	Usually us-east-1; use the destination signing region.

Use Backup settings for connection settings. For changes to storage identity, protection level or bucket set, create a new backup and retain access to old recovery points.

7. Monitor export and preserve recovery information

Select the backup and Refresh. States distinguish Starting backup, Copying files, Checking source files, Creating recovery point and Up to date — watching for changes. Check worker heartbeat, errors/retries, saved/waiting counters and latest completed checkpoint. Worker unavailable needs investigation at the selected node. Inventory continues if notification registration needs attention.

Pause export finishes current work before stopping; Resume export continues. Uploaded objects become part of supported recovery only when the next full catalogue finishes. A copy acknowledgment or quiet queue does not prove physical tape completion or verification.

1. Choose Export recovery bundle.
2. Use an encryption password of at least 16 characters and download the bundle.
3. Keep bundle, password and archive access information outside the primary cluster; store password separately. Protect them as credentials.
4. Refresh recovery information after configuration changes and rehearse import on an isolated replacement.

8. Restore a checkpoint

1. Prepare a working replacement S3 service and credentials able to create/read/write recovery buckets. WORM requires supported protection operations.
2. Choose Restore from archive and a completed checkpoint by its UTC date; the original identifier appears in brackets. Load more if needed.
3. Leave Source bucket empty for all protected buckets, or restrict by bucket, literal prefix or exact object key. Prefix is not a wildcard. Browse saved generations for older copies; preview is limited to 1,000 matches.
4. Enter destination HTTPS endpoint, credentials and private CA chain where needed. Use a separate recovery bucket prefix, such as recovery-. A single target bucket requires one source bucket.
5. Keep replacement of existing objects disabled unless needed for unprotected data. WORM recovery requires new buckets regardless of overwrite settings.
6. Create recovery preview; inspect counts, conflicts, unavailable generations and protection information before starting.
7. Start the reviewed job and monitor progress, failures, pause/resume or cancellation.
8. Verify bytes, retention and legal holds. Review bucket-configuration preview before applying ownership and permissions. Original ownership can remove the recovery account’s access; lifecycle expiration stays disabled during configuration recovery.
9. Restore client routing and SMB gateway/share configuration after content and permissions pass validation.

Use new bucket names rather than deleting WORM buckets or weakening retention. Older checkpoints without verified Object Lock metadata are refused. Independently restored objects need not retain original S3 version identifiers.

After loss of the primary installation

1. Prepare a working replacement eEKAS installation and S3 service, worker connectivity and staging disk.
2. Choose Import recovery bundle; provide the original archive identifier, encrypted bundle smaller than 16 MB and password.
3. The imported configuration is recovery-only with export disabled. Verify archive connection details and keep export disabled during recovery.
4. Preview and restore the checkpoint, recover reviewed bucket settings, validate namespace/permissions and cut over clients.

The bundle is not a standalone generic-server restore application. Slow/offline media may require long recall waits; obtain and test the provider’s physical-media recovery procedure.

9. Troubleshooting and acceptance

Symptom	Check / action
No empty destination	Create a fresh S3 service; do not join a populated namespace.
Invitation rejected	Expiry/use, REST API, management address/18443, routing and certificate trust.
Behind, error or unknown	Both sites, gateways, endpoints, certificates, capacity, synchronization window and connectivity.
Endpoint remains down	Recover S3 gateways; inspect IP Group/virtual endpoint health and reported errors.
No worker heartbeat	Selected node, running worker, source/archive access and staging space.
Notification errors / old checkpoint	Registration, reachability, inventory lag, full catalogue progress and backlog. Upload and checkpoint publication are separate.
Empty bucket-search page	Continue if Next page is available; bounded search may examine only part of the catalogue.
Restore conflicts	New names, target protection support, permissions, Object Lock metadata and unavailable generations.
SMB users cannot see restored data	Gateway endpoint, bucket/path mapping, credentials, shares and identity configuration.

SMB data enters protection only after the gateway commits an S3 object. Open/buffered files, SMB share definitions, identity services and distributed SMB locks are not covered by object replication alone.

• Test dedicated buckets: small/large objects, multipart data, metadata, range reads, bytes and versions.
• For active-active, verify both directions, overwrites, deletions and same-key conflicts with versioning enabled.
• Test retention/legal holds and recovery without weakening protected source data.
• Interrupt connectivity, make controlled writes, reconnect and verify catch-up.
• Rehearse fencing, promotion, client routing and failback in isolation; test local gateway recovery separately.
• Export/import a bundle and restore a completed checkpoint into new buckets; validate ownership and SMB user access.
• Measure backfill, full catalogue duration, worker memory/disk, backlog and restore time for representative counts and object sizes.

The evaluation covered a 1,000-object baseline and selected native replication, WORM and interruption cases. Million-object capacity, total-cluster recovery/failback, complete SMB operation and new external mirror acceptance remain to be demonstrated. Set RPO/RTO using measured deployment results.

Daily: check every native site, export heartbeat, latest completed checkpoint, errors and alert delivery. After changes: verify scope, future-bucket coverage, credentials, trust, routing and bundle currency. Regularly: test reads, protection and recovery, and review capacity. Before decommissioning: retain a verified recovery alternative.

Protect SMB shares backed by S3 WORM

See SMB Shares with S3 WORM to expose a COMPLIANCE-protected bucket through SMB. Replication and backup protect the S3 objects committed by the gateway. Include share, identity and gateway configuration, and verification of destination retention, in your recovery plan.

REST API and automation

For automated status queries and administrative tasks, read REST API – Getting Started and Automation. The API reference and S3 workflows guide in the download package describe replication and backup operations. The prerequisites and checks in this chapter also apply to API calls.

Customer registries and billing after a site change

Native object replication and content recovery do not automatically copy customer registries, tariffs or historical billing ledgers. Export customer metadata and approved billing reports and retain them outside the primary installation.

Register an existing replicated or recovered user and bucket through POST /api/v1/object-storage/customers/adopt rather than recreating data or access keys. Verify the UID against the existing bucket owner. Adoption records the agreed quota without changing the enforced quota.

Cross-installation billing is configured separately and needs an unambiguous mapping of accounting zones and pricing owner. A reachable peer or synchronized object state does not confirm complete traffic collection. Check coverage before approving charges; missing counters do not mean zero usage. A site change does not trigger automatic accounting failover.

Traffic during backup and recovery

Managed archive reads use system credentials at the source and are excluded from customer gateway usage. Caller-supplied source credentials must also be system credentials. External uploads and recovery requests may still be billed at their destination when its bucket is registered for billing and ordinary customer credentials are used.

Detailed API workflows

S3 Replication, Backup and Recovery via REST API covers invitations, preflight, policies, backfill, backup catalogues and WORM recovery. API jobs for setup or bundles and restore-worker jobs for preview and recovery must be tracked separately. For adoption and consolidated billing, read S3 Customer Management, Usage and Billing via REST API.