bedrock
tenant layer

CloudNativePG

CloudNativePG 1.27.0 runs PostgreSQL 17.4. The operator lives in cnpg-system and manages one Cluster resource per database namespace.

The clusters

Settingcluster-1cluster-2cluster-3
RolePrimaryLive replicaDelayed replica
Instances222
Created byinitdb (database app, owner appuser)Recovery from cluster-1’s backupsRecovery from cluster-1’s backups
ReplicationSynchronous inside the clusterStreams from cluster-1Streams from cluster-1, 5 minutes behind
Volume10 GiB on longhorn10 GiB10 GiB
Priority classbedrock-databedrock-databedrock-data

Each instance requests 256 MiB and 100m of CPU, with limits of 1 GiB and 1 CPU.

Hardening

SettingEffect
enableSuperuserAccess: falseThe postgres user cannot log in over the network. Applications use their own users.
pg_hba: hostnossl all all all rejectPlaced before CloudNativePG’s default rules, it refuses every connection that is not TLS
log_statement: ddlEvery schema change, including DROP, is written to the log and shipped to Loki
Replicas connect with sslmode: verify-caThe replica checks it is talking to the real cluster-1
Replicas read backups with a read-only keyA compromised replica cannot change or delete cluster-1’s backups

Synchronous replication

cluster-1 uses quorum-based synchronous replication:

synchronous:
  method: any
  number: 1
  dataDurability: preferred

A commit returns once at least one standby instance has written it. preferred means that if no standby is up, the primary carries on alone instead of blocking every write. cluster-2 and cluster-3 are separate clusters and are not part of this.

Replica clusters

cluster-2 and cluster-3 start with bootstrap.recovery from cluster-1’s backup bucket, then switch to streaming from cluster-1-rw.cluster-1.svc with replica.enabled: true. cluster-3 adds minApplyDelay: 5m. Because the delay is incompatible with CloudNativePG’s promotion token, cluster-3 is promoted by setting replica.enabled: false.

The replicas need cluster-1’s CA and replication certificate in their own namespaces. Kyverno copies them and keeps them current (see Kyverno).

Poolers

Each cluster has a PgBouncer Pooler with more than one instance in transaction mode, max_client_conn 1000 and default_pool_size 25, and a disruption budget that keeps one running. The poolers on cluster-2 and cluster-3 stay ready but only accept writes once their cluster is promoted.

postgres-rw.application.svc is an ExternalName service pointing at the active pooler. Promotion changes only where it points.

Backups

SettingValue
DestinationOne Garage bucket per cluster, s3://cluster-N-backups/wals
WAL archivingContinuous, gzip
Base backupsEvery 6 hours (0 0 */6 * * *), gzip
Retention30 days
First backupcluster-1-initial, which must complete before the replicas can be created

The S3 client needs AWS_REQUEST_CHECKSUM_CALCULATION and AWS_RESPONSE_CHECKSUM_VALIDATION set to when_required to work with Garage, and AWS_DEFAULT_REGION set to garage.

To recover data from these backups, see restore a database to a point in time.

Spread

Pod anti-affinity is preferred, not required. If a node dies, its instance moves onto a surviving node instead of staying Pending, which keeps the cluster available at the cost of instances sharing a node. CNPGStackedInstances fires when that happens, because instances never move back on their own.

Monitoring

Every cluster has enablePodMonitor: true, so Prometheus scrapes each instance on 9187. Alerts cover replication lag, instance health, WAL archiving, backup age and sudden spikes of deleted or updated rows.