GEO Redundancy support for WRC

Overview

Geo-redundancy refers to the practice of:

Placing physical servers in geographically diverse data centers to safeguard against catastrophic events and natural disasters.
Replicating business data to another physical location elsewhere in the world as a safety net against regional outages.

Implementation details for Studio Conductor

Deploy Conductor into two regions with Primary and Secondary (Stand by) Cluster for replication.
Clusters should be able to communicate with each other. Ensure proper configuration provided in the override values yaml.
Architecture for the postgresql should be defined as replication.
Replication Attributes should be added in the values.yaml to provide the support for the replication.
Provide the security related details for the securing the data replication across cluster:
- A valid certificate available and accessible to the cluster.
- Update the configuration with the certificates and key details.

WRC GEO-Redundancy WRC GEO-Redundancy Implementation

Primary Configuration Details for Postgresql

The following example provides configuration details for Postgresql. These values need to be added or updated in the override values .yaml file. This includes:

Exposing the service via nodeport.
Adding pgHba configuration. Multiple rows for host and hostssl can be added if you are installing with security enabled.
Setting up the architecture as Replication with one slave as the replica.
Enabling the TLS flag and adding the secret details if you are installing with security enabled.


postgresql:
  primary:
    service:
      ports:
        postgresql: 5432
      type: NodePort
      nodePorts:
        postgresql: <nodeport - range 30000-32767> # Port to expose for PrimaryCluster Machine
    pgHbaConfiguration: |-
      host         all             all        127.0.0.1/32     md5 # Required for Local ip
      host     replication       postgres     x.x.x.x/x        md5 # Provide Machine Specific IP range for pod/service
      host     replication       postgres     x.x.x.x/x        md5 # Provide the Second Machine details with subnet
      hostssl  replication     postgres       x.x.x.x/x        md5 # Provide the Second Machine details with subnet 
      host     cloudify_db     cloudify       ::1/128          md5
      hostssl   all             all           0.0.0.0/0        reject # Required to reject all the connection request except the ones which is configured in the config
      local        all             all                         trust
  architecture: replication
  replication:
    synchronousCommit: "on"
    numSynchronousReplicas: 1
    slaveReplicas: 1
  readReplicas:
    replicaCount: 5
  tls:
    enabled: false
    autoGenerated: false
    preferServerCiphers: true
    certificatesSecret: ""  # Provide secret name For e.g postgres-certs
    certFilename: "" # Provide certificate filename for e.g tls.crt
    certKeyFilename: "" # Provide certkey filename for e.g tls.key
    certCAFilename: "" # Provide cacert filename for e.g root.crt

# all the db-using services need to be instructed to connect to the primary
db:
  host: "postgresql-primary"

Secondary Configuration Details for Postgresql

The following example provides secondary configuration details for Postgresql. These values need to be added or updated in the override values .yaml file. This includes:

Adding the standby section to make it the standby cluster.
Exposing the service via nodeport.
Adding pgHba configuration. Multiple rows for host and hostssl can be added if you are installing with security enabled. For more information, see PostgreSQL.
Setting up the architecture as Replication with one slave as the replica.
Enabling the TLS flag and adding the secret details if you are installing with security enabled.

postgresql:
  primary:
    service:
      ports:
        postgresql: 5432
      type: NodePort
      nodePorts:
        postgresql: <nodeport - range 30000-32767> # Port to expose for Secondary Cluster Machine
    standby:
      enabled: true
      primaryHost: <Primary Host IP> # IP Address of the Primary Cluster Machine
      primaryPort: <Primary Host Nodeport> # Exposed Port of the Primary Cluster Machine
    # We might to have provide multiple entries for pghba configuration. 
    pgHbaConfiguration: |-
      host         all             all        127.0.0.1/32     md5 # Required for Local ip
      host     replication       postgres     x.x.x.x/x        md5 # Provide Machine Specific IP range for pod/service
      host     replication       postgres     x.x.x.x/x        md5 # Provide the Second Machine details with subnet
      hostssl  replication     postgres       x.x.x.x/x        md5 # Provide the Second Machine details with subnet 
      host     cloudify_db     cloudify       ::1/128          md5
      hostssl   all             all           0.0.0.0/0        reject # Required to reject all the connection request except the ones which is configured in the config
      local        all             all                         trust

  architecture: replication
  replication:
    synchronousCommit: "on"
    numSynchronousReplicas: 1
    slaveReplicas: 1
  readReplicas:
    replicaCount: 2
  tls:
    enabled: false
    autoGenerated: false
    preferServerCiphers: true
    certificatesSecret: ""  # Provide secret name For e.g postgres-certs
    certFilename: "" # Provide certificate filename for e.g tls.crt
    certKeyFilename: "" # Provide certkey filename for e.g tls.key
    certCAFilename: "" # Provide cacert filename for e.g root.crt
    

# all the db-using services need to be instructed to connect to the primary
db:
  host: "postgresql-primary"

Primary Configuration Details for Seaweedfs

The following example provides configuration details for Seaweedfs:

filer:
    replicaCount: 1
    service:
      type: NodePort
      nodePorts:
        http: <PortNumber Range 30000-32767>
        grpc: <PortNumber Range 30000-32767>
  security:
    enabled: false
    mTLS:
      enabled: false
      ca: ""
      existingCASecret: "<Secret Name>"
      master:
        cert: ""
        key: ""
        existingSecret: "<Secret Name>"
      volume:
        cert: ""
        key: ""
        existingSecret: "<Secret Name>"
      filer:
        cert: ""
        key: ""
        existingSecret: "<Secret Name>"
      client:
        cert: ""
        key: ""
        existingSecret: "<Secret Name>"

Custom failover Geo-Redundancy Config

To implement the failover and auto switch over, a configuration job is available that performs the following tasks:

Starts the replication of the Seaweedfs data between clusters.
Monitor the Postgresql DB for Primary and Secondary Cluster DB.
If Primary goes down, then Secondary will promote itself to Primary and user can move to the secondary server to continue.
When Original Primary comes up again, then it will demote itself and starts data replication from new primary (original secondary) cluster.

A sample entry for the geo-redundancy custom failover job is shown below. In this example:

If Studio Conductor is deployed with geo-redundant functionality then the enabled attribute should be set as true
primaryHostAddress should be the ip address of the Primary cluster (where conductor is going to be installed)
primaryHostPort should be the exposed port the Primary cluster.
primaryHostAddress should be the ip address of the Primary cluster (where conductor is going to be installed)
secondaryHostPort should be the exposed port the Primary cluster.
original_role: Should be Primary for Primary Cluster and Secondary for Standby Cluster
current_role: Should be Primary for Primary Cluster and Secondary for Standby Cluster
Other attributes are shown with default values, refer to the values.yaml for more information.


geo_redundancy_custom_failover:
  enabled: false
  image: (registry-link)/geo-redundancy-custom-failover
  env:
     primaryHostAddress: "<PrimaryHostIP>"
     primaryHostPort: "<PrimaryPort>"
     secondaryHostAddress: "<SecondaryHostIP>"
     secondaryHostPort: "<SecondaryPort>"
     maxRetries: "5"
  configmapValues: |
        {
         "original_role": <Role "Primary" or "Secondary">,
         "current_role" : <Role "Primary" or "Secondary">
        }