AVAILABILITY & RECOVERY
Keep services recoverable. Make the trade-offs visible.
Halo combines WAL-based replication, queryable standbys, failover orchestration, and logical and physical recovery paths. Commit policy, topology, distance, and operating discipline determine the result.
Four working layers support continuity
Replication moves change, orchestration manages database roles, a service address directs traffic, and backup provides an independent recovery path.
Service address / VIP
Traffic reaches the current read/write role.
Primary and readable standbys
Physical WAL streams and logical publication/subscription serve different replication needs.
Halo Shield
clustermgr on each node · etcd quorum · health · role lifecycle · fencing.
RMAN2 and WAL archive
Backup, validation, restore, PITR, or standby preparation.
Halo Shield, a separate keepalived reference pattern, replication, and RMAN2 serve different purposes and are not interchangeable layers.
Choose the replication path by purpose
| Approach | Appropriate use | Important boundary |
|---|---|---|
| Physical streaming | Close standby copies, readable replicas, and failover topologies | Commit policy and reachability determine latency and data exposure |
| Logical publication/subscription | Selective data movement | Not a complete substitute for physical HA or backup |
| Cascaded streaming | Extend a topology through an intermediate standby | Each link adds another lag and operating point |
| Replication slots | Retain required WAL for a consumer | Delayed consumers can increase retained storage and need monitoring |
One primary, multiple replication purposes
Grouped replication defines a main group and up to four additional groups, each with its own synchronous level. An additional group cannot be stronger than the main group.
Commit origin
Writes generate WAL for every configured replication path.
Local availability
Synchronous and potential synchronous members can be observed per group.
Remote recovery roles
Up to four groups can use independent, no-stronger commit levels.
Documented synchronous strength runs remote_apply → on → remote_write → local/off. RPO 0 is a topology objective only while the required synchronously applied replica is reachable; it is not a universal SLA.
Halo Shield manages the database role lifecycle
Reference architectureDatabase nodes plus etcd
The manual illustrates one primary, two replicas, clustermgr on each database node, and a VIP that moves with the primary role. That database-node layout is a reference topology. Separately, the documented etcd quorum requires at least three members and an odd member count for leader election.
Consensus and fencingAvoid dual primary roles
etcd maintains role and leader-lock state. A node that cannot renew the leader lock can stop its database, and minority-side availability is sacrificed to prevent split brain. Watchdog can provide an additional fencing layer.
Watchdog policyoff · automatic · required
Watchdog can restart a node if clustermgr stops making progress. In automatic mode, Shield can continue with a warning when no watchdog device is available; required mode rejects clustermgr startup without the device. The manual describes both software and hardware watchdog paths.
Automatic and manual transitionsFailure and maintenance paths
Shield supports automatic failover when detection conditions are met, operator-directed failover, and planned switchover. A planned switchover can cause a brief interruption; the site does not claim zero downtime.
Maintenance controlsOperate a protected cluster
Operators can pause and resume automatic failover, update dynamic DCS configuration, reinitialize a replica, restart or reload selected members, and coordinate rolling maintenance.
Operational evidenceSee what changed
Status and lag, tree topology, transition history, current connection information, component logs, etcd health, and REST responses for primary, replica, cluster, and metrics support investigation.
Alternative keepalived patternSeparate two-node design
The manual also documents a two-node streaming-replication pattern with keepalived and a VIP. It is distinct from Shield and should not be assumed to share the same quorum or fencing behavior.
Recover at the right level
Protection taxonomyFull/incremental · logical/physical · hot/cold
A complete plan can include data, objects, users, privileges, and relevant environment information. Hot backup lets the database continue to serve reads and writes; cold backup requires shutdown.
Logical backup and restorepg_dump · pg_dumpall · pg_restore
Coverage can be a database, schema, or table, or cluster-wide databases and global objects such as roles and tablespaces. Plain and archive formats support different workflows. The documented compatibility-object scope includes packages, type bodies, DBLINK, and synonyms with selective tool behavior. Logical methods are useful across versions and platforms but can be slower at scale and do not provide PITR.
RMAN2 physical recoveryFULL · DELTA · PAGE · PTRACK
RMAN2 documents full and incremental modes, including page-level change tracking through PTRACK, plus WAL streaming, zlib or pglz compression, parallel backup/restore/verification, integrity validation, and skipping never-written blocks. Physical backup requires the database to run in archive mode.
Point-in-time targetsTime · LSN · transaction · restore point · timeline
Recovery can stop at a documented point rather than only at the end of available WAL. The archive and backup chain must make that target reachable.
Restore, validate, and catch upNew-directory workflows
A restore can target a new directory for non-disruptive validation or standby preparation. FULL, DELTA, and PTRACK modes can support replica catch-up; PAGE mode is not documented for that catch-up path.
Lifecycle managementCatalogs, retention, and WAL handling
Backup catalogs, history, retention and TTL policies, WAL archive push/fetch/verification, named recovery targets, and backup validation support an auditable recovery program.
Validate the failure modes that matter
- Commit latency and data exposure at each synchronous level
- Node, network, quorum, and service-address failures
- Replica lag and freshness of reads served from standbys
- WAL retention while consumers are delayed
- Planned switchover, unexpected failover, and former-primary rejoin
- Backup validation and timed restore or PITR rehearsals
- Monitoring and decision ownership during degraded operation
Halo provides the mechanisms. The deployment must establish its own topology, commit policy, recovery sequence, and measured RPO/RTO.