Installing a Compact Cluster (3 Nodes) Manually
Compact Cluster (3 Nodes) Manual Installation
A three nodes cluster provides a highly available active-active configuration where all Studio Conductor services run on each node.
These instructions explain how to install a three nodes cluster without using Cluster Manager. To use Cluster Manager to automate the installation process, see Installing a Three Nodes Cluster with Cluster Manager.
If air-gapped opertation of the site map is required after manual installation, see Manual Installation Requirements for Air-Gapped Operation for additional configuration instructions. (Note: these instructions appear at the end of the instructions for installing a fully distributed (9 nodes) cluster but are also applicable to the 3 node installation).
Note: This configuration does not necessarily mean that all communication is internal within each node. For example, the database high availability setup is such that at any given time one node is active while the other nodes are in synchronous replica and async replica. With this configuration, all the managers in each node communicate with the same active database node, regardless of whether it is local or remote.
Compact Cluster Network Architecture
Installation Overview
Setting up a three nodes cluster involves the following steps:
- Verifying that your environment meets the basic prerequisites.
- Generating the certificates.
- Opening the required ports.
- Running the Install program for the database, broker, and manager on all three nodes.
- Completing Day 2 requirements.
Prerequisites
Review the following prerequisites to make sure your system supports this configuration. For general guidelines, see Sizing Guidelines.
Operating System
The recommended operating system for a three nodes cluster is CentOS 7.9.
Update to this system if required and, after downloading, log in as root and update the base image packages using the following command:
yum update -y
reboot
Note: If an internet connection is not available, an alternate method will need to be used to update the base image packages.
Configuration requirements
The following configuration settings should be available prior to installation:
- Public and private IP settings
- A configurable host name
- Administrator privileges (e.g. sudo permissions)
- All nodes should be on the same network and, if there is a firewall or security group, used ports are open and not blocking relevant services.
Sizing Guidelines
Resource | Recommended |
---|---|
vCPUs | 8 |
RAM | 16GB |
Storage | 64GB |
Preparing for Installation
The following steps are required prior to running the AIO installation:
- Install the manager RPM file on your system.
- Upload the license file to all the nodes in the cluster.
- Install the required Python packages.
- Generate cluster certificates.
- Open the required ports
Installing the Manager RPM
The RPM file contains all the components and dependencies required to run the installation process and is available on Wind River Delivers, Wind River’s software portal. For detailed instructions on accessing Wind River Delivers and downloading the file, see the Wind River Installation and Licensing Guide
To install the Manger RPM, log in as root and enter:
yum install -y $HOME/cloudify-manager-install-22.11-ga.el7.x86_64.rpm
Uploading the License File to each Nodes
Copy the license file you received from Wind River to each of the nodes and document the path. You will need to enter this path when you update the config.yaml file.
Installing Required Packages
Additional Python packages are required to support the Manager. As root, enter the following:
yum install -y unzip rsync python-setuptools python-backports python-backports-ssl_match_hostname
Generating Certificates
To allow communication across the cluster, certifications need to generated and copied to each host in the cluster. For additional information about certificates see Certificates Overview.
To generate test certificates using cty_manager’s built in commands, perform the following:
On the same host in the cluster, enter:
cfy_manager generate-test-cert -s <node-1-fqdn>,<node-1-private-ip>,<node-1-public-ip> cfy_manager generate-test-cert -s <node-2-fqdn>,<node-2-private-ip>,<node-2-public-ip> cfy_manager generate-test-cert -s <node-3-fqdn>,<node-3-private-ip>,<node-3-public-ip>
Copy the relevant certificates/keys from the ‘$HOME/.cloudify-test-ca/’ on host where the certificates were generated to the other hosts in the cluster.
The following shows an example of the test certificates:
Example certificates
cfy_manager generate-test-cert -s hostname1.example.com,192.0.2.1,203.0.113.1
cfy_manager generate-test-cert -s hostname2.example.com,192.0.2.2,203.0.113.2
cfy_manager generate-test-cert -s hostname3.example.com,192.0.2.3,203.0.113.3
Open Ports for Network Access
For proper network communication, open the posts listed below on all three nodes.
firewall-cmd --permanent --add-port=22/tcp
firewall-cmd --permanent --add-port=443/tcp
firewall-cmd --permanent --add-port=2379/tcp
firewall-cmd --permanent --add-port=2380/tcp
firewall-cmd --permanent --add-port=5432/tcp
firewall-cmd --permanent --add-port=8008/tcp
firewall-cmd --permanent --add-port=8009/tcp
firewall-cmd --permanent --add-port=4369/tcp
firewall-cmd --permanent --add-port=5672/tcp
firewall-cmd --permanent --add-port=25672/tcp
firewall-cmd --permanent --add-port=35672/tcp
firewall-cmd --permanent --add-port=15672/tcp
firewall-cmd --permanent --add-port=61613/tcp
firewall-cmd --permanent --add-port=1883/tcp
firewall-cmd --permanent --add-port=15674/tcp
firewall-cmd --permanent --add-port=15675/tcp
firewall-cmd --permanent --add-port=15692/tcp
firewall-cmd --permanent --add-port=5671/tcp
firewall-cmd --permanent --add-port=22000/tcp
firewall-cmd --permanent --add-port=53333/tcp
firewall-cmd --permanent --add-port=25671/tcp
firewall-cmd --permanent --add-port=15671/tcp
firewall-cmd --reload
firewall-cmd --list-ports
Installing the Database
Once the preliminary installation tasks are complete, log in as root and run the following steps on all three nodes.
- On each node, use a text editor to create the file /etc/cloudify/db_config.yaml and enter your specific network parameters. Update the fields shown below by replacing the values marked in <> with values for your network.
Note: This must be performed sequentially on each node.
# /etc/cloudify/db_config.yaml
manager:
private_ip: '<private-ip>'
public_ip: '<public-ip>'
postgresql_server:
postgres_password: '<secure-password-like-string>'
cert_path: '<this-node-local-certificate-path>'
key_path: '<this-node-private-key-path>'
ca_path: '<local-ca-certificate-path>'
cluster:
nodes:
<node-1-host-name>:
ip: '<node-1-private-ip-address>'
<node-2-host-name>:
ip: '<node-2-private-ip-address>'
<node-3-host-name>:
ip: '<node-3-private-ip-address>'
etcd:
cluster_token: '<cluster-token>'
root_password: '<secure-password-like-string>'
patroni_password: '<secure-password-like-string>'
patroni:
rest_user: '<rest-user->'
rest_password: '<secure-password-like-string>'
postgres:
replicator_password: '<secure-password-like-string>'
prometheus:
credentials:
username: '<username>'
password: '<secure-password-like-string>'
cert_path: '<this-node-local-certificate-path>'
key_path: '<this-node-local-private-key-path>'
ca_path: '<local-ca-certificate-path>'
services_to_install:
- database_service
- monitoring_service
On each node, enter the following to run the installation process for the database.
cfy_manager install -c /etc/cloudify/db_config.yaml
Example process for Node 1:
# /etc/cloudify/db_config.yaml
manager:
private_ip: '192.0.2.1'
public_ip: '203.0.113.1'
postgresql_server:
postgres_password: 'strongadminpassword'
cert_path: '/root/.cloudify-test-ca/hostname1.example.com.crt'
key_path: '/root/.cloudify-test-ca/hostname1.example.com.key'
ca_path: '/root/.cloudify-test-ca/ca.crt'
cluster:
nodes:
hostname1:
ip: '192.0.2.1'
hostname2:
ip: '192.0.2.2'
hostname3:
ip: '192.0.2.3'
etcd:
cluster_token: 'clustertokennode1'
root_password: 'strongrootpasswordnode1'
patroni_password: 'strongpatronipasswordnode1'
patroni:
rest_user: 'restusername'
rest_password: 'strongrestpasswordnode1'
postgres:
replicator_password: 'strongreplicatorpasswordnode1'
prometheus:
credentials:
username: 'prometheususername'
password: 'strongprometheuspasswordnode1'
cert_path: '/root/.cloudify-test-ca/hostname1.example.com.crt'
key_path: '/root/.cloudify-test-ca/hostname1.example.com.key'
ca_path: '/root/.cloudify-test-ca/ca.crt'
services_to_install:
- database_service
- monitoring_service
cfy_manager install -c /etc/cloudify/db_config.yaml
Example process for Node 2:
# /etc/cloudify/db_config.yaml
manager:
private_ip: '192.0.2.2'
public_ip: '203.0.113.2'
postgresql_server:
postgres_password: 'strongadminpassword'
cert_path: '/root/.cloudify-test-ca/hostname2.example.com.crt'
key_path: '/root/.cloudify-test-ca/hostname2.example.com.key'
ca_path: '/root/.cloudify-test-ca/ca.crt'
cluster:
nodes:
hostname1:
ip: '192.0.2.1'
hostname2:
ip: '192.0.2.2'
hostname3:
ip: '192.0.2.3'
etcd:
cluster_token: 'clustertokennode2'
root_password: 'strongrootpasswordnode2'
patroni_password: 'strongpatronipasswordnode2'
patroni:
rest_user: 'restusername'
rest_password: 'strongrestpasswordnode2'
postgres:
replicator_password: 'strongreplicatorpasswordnode2'
prometheus:
credentials:
username: 'prometheususername'
password: 'strongprometheuspasswordnode2'
cert_path: '/root/.cloudify-test-ca/hostname2.example.com.crt'
key_path: '/root/.cloudify-test-ca/hostname2.example.com.key'
ca_path: '/root/.cloudify-test-ca/ca.crt'
services_to_install:
- database_service
- monitoring_service
cfy_manager install -c /etc/cloudify/db_config.yaml
Example process for Node 3:
# /etc/cloudify/db_config.yaml
manager:
private_ip: '192.0.2.3'
public_ip: '203.0.113.3'
postgresql_server:
postgres_password: 'strongadminpassword'
cert_path: '/root/.cloudify-test-ca/hostname3.example.com.crt'
key_path: '/root/.cloudify-test-ca/hostname3.example.com.key'
ca_path: '/root/.cloudify-test-ca/ca.crt'
cluster:
nodes:
hostname1:
ip: '192.0.2.1'
hostname2:
ip: '192.0.2.2'
hostname3:
ip: '192.0.2.3'
etcd:
cluster_token: 'clustertokennode3'
root_password: 'strongrootpasswordnode3'
patroni_password: 'strongpatronipasswordnode3'
patroni:
rest_user: 'restusername'
rest_password: 'strongrestpasswordnode3'
postgres:
replicator_password: 'strongreplicatorpasswordnode3'
prometheus:
credentials:
username: 'prometheususername'
password: 'strongprometheuspasswordnode3'
cert_path: '/root/.cloudify-test-ca/hostname3.example.com.crt'
key_path: '/root/.cloudify-test-ca/hostname3.example.com.key'
ca_path: '/root/.cloudify-test-ca/ca.crt'
services_to_install:
- database_service
- monitoring_service
cfy_manager install -c /etc/cloudify/db_config.yaml
Installing the Broker
Once the database is installed on each node, log in as root and run the following steps on each of the three nodes to install the broker.
- On each node, use a text editor to create the file /etc/cloudify/rabbitmq_config.yaml and enter your specific network parameters. Update the fields shown below by replacing the values marked in <> with values for your network.
Note: This must be performed sequentially on each node.
# /etc/cloudify/rabbitmq_config.yaml
manager:
private_ip: '<private-ip>'
public_ip: '<public-ip>'
rabbitmq:
username: '<username>'
password: '<secure-password-like-string>'
cert_path: '<this-node-local-certificate-path>'
key_path: '<this-node-local-private-key-path>'
ca_path: '<local-ca-certificate-path>'
cluster_members:
<node-1-host-name>:
networks:
default: '<node-1-private-ip>'
<node-2-host-name>:
networks:
default: '<node-2-private-ip>'
<node-3-host-name>:
networks:
default: '<node-3-private-ip>'
nodename: '<this-node-hostname>'
join_cluster: '<node-1 hostname; **should be left blank on node-1**>'
erlang_cookie: '<secure-password-like-string>'
prometheus:
credentials:
username: '<username>'
password: '<secure-password-like-string>'
cert_path: '<this-node-local-certificate-path>'
key_path: '<this-node-local-private-key-path>'
ca_path: '<local-ca-certificate-path>'
services_to_install:
- queue_service
- monitoring_service
On each broker node, enter the following to run the installation process.
cfy_manager install -c /etc/cloudify/rabbitmq_config.yaml
Example process for Node 1:
# /etc/cloudify/rabbitmq_config.yaml
manager:
private_ip: '192.0.2.1'
public_ip: '203.0.113.1'
rabbitmq:
username: 'rabbitmqadmin'
password: 'strongadminpasswordnode1'
cert_path: '/root/.cloudify-test-ca/hostname1.example.com.crt'
key_path: '/root/.cloudify-test-ca/hostname1.example.com.key'
ca_path: '/root/.cloudify-test-ca/ca.crt'
cluster_members:
hostname1:
networks:
default: '192.0.2.1'
hostname2:
networks:
default: '192.0.2.2'
hostname3:
networks:
default: '192.0.2.3'
nodename: 'hostname1'
erlang_cookie: 'cookiename'
prometheus:
credentials:
username: 'prometheususername'
password: 'strongprometheuspasswordnode1'
cert_path: '/root/.cloudify-test-ca/hostname1.example.com.crt'
key_path: '/root/.cloudify-test-ca/hostname1.example.com.key'
ca_path: '/root/.cloudify-test-ca/ca.crt'
services_to_install:
- queue_service
- monitoring_service
cfy_manager install -c /etc/cloudify/rabbitmq_config.yaml
Example process for Node 2:
# /etc/cloudify/rabbitmq_config.yaml
manager:
private_ip: '192.0.2.2'
public_ip: '203.0.113.2'
rabbitmq:
username: 'rabbitmqadmin'
password: 'strongadminpasswordnode2'
cert_path: '/root/.cloudify-test-ca/hostname2.example.com.crt'
key_path: '/root/.cloudify-test-ca/hostname2.example.com.key'
ca_path: '/root/.cloudify-test-ca/ca.crt'
cluster_members:
hostname1:
networks:
default: '192.0.2.1'
hostname2:
networks:
default: '192.0.2.2'
hostname3:
networks:
default: '192.0.2.3'
nodename: 'hostname2'
join_cluster: 'hostname1'
erlang_cookie: 'cookiename'
prometheus:
credentials:
username: 'prometheususername'
password: 'strongprometheuspasswordnode2'
cert_path: '/root/.cloudify-test-ca/hostname2.example.com.crt'
key_path: '/root/.cloudify-test-ca/hostname2.example.com.key'
ca_path: '/root/.cloudify-test-ca/ca.crt'
services_to_install:
- queue_service
- monitoring_service
cfy_manager install -c /etc/cloudify/rabbitmq_config.yaml
Example process for Node 3:
# /etc/cloudify/rabbitmq_config.yaml
manager:
private_ip: '192.0.2.3'
public_ip: '203.0.113.3'
rabbitmq:
username: 'rabbitmqadmin'
password: 'strongadminpasswordnode3'
cert_path: '/root/.cloudify-test-ca/hostname3.example.com.crt'
key_path: '/root/.cloudify-test-ca/hostname3.example.com.key'
ca_path: '/root/.cloudify-test-ca/ca.crt'
cluster_members:
hostname1:
networks:
default: '192.0.2.1'
hostname2:
networks:
default: '192.0.2.2'
hostname3:
networks:
default: '192.0.2.3'
nodename: 'hostname3'
join_cluster: 'hostname1'
erlang_cookie: 'cookiename'
prometheus:
credentials:
username: 'prometheususername'
password: 'strongprometheuspasswordnode3'
cert_path: '/root/.cloudify-test-ca/hostname3.example.com.crt'
key_path: '/root/.cloudify-test-ca/hostname3.example.com.key'
ca_path: '/root/.cloudify-test-ca/ca.crt'
services_to_install:
- queue_service
- monitoring_service
cfy_manager install -c /etc/cloudify/rabbitmq_config.yaml
Installing the Manager
Once the broker is installed, log in as root and run the following steps to install the manager on each nodes.
- On each node, use a text editor to create the file /etc/cloudify/manager_config.yaml and enter your specific network parameters. Update the fields shown below by replacing the values marked in <> with values for your network.
Note: This must be performed sequentially on each node.
# /etc/cloudify/manager_config.yaml
manager:
private_ip: '<private-ip>'
public_ip: '<private-ip>'
security:
ssl_enabled: true
admin_password: '<secure-password-like-string>'
cloudify_license_path: '<cloudify-license-path>'
postgresql_server:
ca_path: '<local-ca-certificate-path>'
cluster:
nodes:
<node-1-host-name>:
ip: '<node-1-private-ip>'
<node-2-host-name>:
ip: '<node-2-private-ip>'
<node-3-host-name>:
ip: '<node-3-private-ip>'
postgresql_client:
server_password: '<secure-password-like-string>'
ssl_enabled: true
networks:
default: '<this-node-private-ip>'
rabbitmq:
username: '<username>'
password: '<secure-password-like-string>'
ca_path: '<local-ca-certificate-path>'
cluster_members:
<node-1-host-name>:
networks:
default: '<node-1-private-ip>'
<node-2-host-name>:
networks:
default: '<node-2-private-ip>'
<node-3-host-name>:
networks:
default: '<node-3-private-ip>'
ssl_inputs:
external_cert_path: '<this-node-local-certificate-path>'
external_key_path: '<this-node-local-private-key-path>'
internal_cert_path: '<this-node-local-certificate-path>'
internal_key_path: '<this-node-local-private-key-path>'
ca_cert_path: '<local-ca-certificate-path>'
external_ca_cert_path: '<local-ca-certificate-path>'
prometheus:
credentials:
username: '<username>'
password: '<secure-password-like-string>'
cert_path: '<this-node-local-certificate-path>'
key_path: '<this-node-local-private-key-path>'
ca_path: '<local-ca-certificate-path>'
services_to_install:
- manager_service
- monitoring_service
- entropy_service
On each node, enter the following to run the manager installation process.
cfy_manager install -c /etc/cloudify/manager_config.yaml
Example manager_config.yaml for Node 1:
# /etc/cloudify/manager_config.yaml
manager:
private_ip: '192.0.2.1'
public_ip: '203.0.113.1'
security:
ssl_enabled: true
admin_password: 'strongadminpasswordnode1'
cloudify_license_path: '/root/cloudify/license.yaml'
postgresql_server:
ca_path: '/root/.cloudify-test-ca/ca.crt'
cluster:
nodes:
hostname1:
ip: '192.0.2.1'
hostname2:
ip: '192.0.2.2'
hostname3:
ip: '192.0.2.3'
postgresql_client:
server_password: 'strongserverpassword'
ssl_enabled: true
networks:
default: '192.0.2.1'
rabbitmq:
username: 'rabbitmqusername'
password: 'strongrabbitmqpassword'
ca_path: '/root/.cloudify-test-ca/ca.crt'
cluster_members:
hostname1:
networks:
default: '192.0.2.1'
hostname2:
networks:
default: '192.0.2.2'
hostname3:
networks:
default: '192.0.2.3'
ssl_inputs:
external_cert_path: '/root/.cloudify-test-ca/hostname1.example.com.crt'
external_key_path: '/root/.cloudify-test-ca/hostname1.example.com.key'
internal_cert_path: '/root/.cloudify-test-ca/hostname1.example.com.crt'
internal_key_path: '/root/.cloudify-test-ca/hostname1.example.com.key'
ca_cert_path: '/root/.cloudify-test-ca/ca.crt'
external_ca_cert_path: '/root/.cloudify-test-ca/ca.crt'
prometheus:
credentials:
username: 'prometheususername'
password: 'strongprometheuspasswordnode1'
cert_path: '/root/.cloudify-test-ca/hostname1.example.com.crt'
key_path: '/root/.cloudify-test-ca/hostname1.example.com.key'
ca_path: '/root/.cloudify-test-ca/ca.crt'
services_to_install:
- manager_service
- monitoring_service
- entropy_service
cfy_manager install -c /etc/cloudify/manager_config.yaml
Example manager_config.yaml for Node 2:
# /etc/cloudify/manager_config.yaml
manager:
private_ip: '192.0.2.2'
public_ip: '203.0.113.2'
security:
ssl_enabled: true
admin_password: 'strongadminpasswordnode2'
cloudify_license_path: '/root/cloudify/license.yaml'
postgresql_server:
ca_path: '/root/.cloudify-test-ca/ca.crt'
cluster:
nodes:
hostname1:
ip: '192.0.2.1'
hostname2:
ip: '192.0.2.2'
hostname3:
ip: '192.0.2.3'
postgresql_client:
server_password: 'strongserverpassword'
ssl_enabled: true
networks:
default: '192.0.2.2'
rabbitmq:
username: 'rabbitmqusername'
password: 'strongrabbitmqpassword'
ca_path: '/root/.cloudify-test-ca/ca.crt'
cluster_members:
hostname1:
networks:
default: '192.0.2.1'
hostname2:
networks:
default: '192.0.2.2'
hostname3:
networks:
default: '192.0.2.3'
ssl_inputs:
external_cert_path: '/root/.cloudify-test-ca/hostname2.example.com.crt'
external_key_path: '/root/.cloudify-test-ca/hostname2.example.com.key'
internal_cert_path: '/root/.cloudify-test-ca/hostname2.example.com.crt'
internal_key_path: '/root/.cloudify-test-ca/hostname2.example.com.key'
ca_cert_path: '/root/.cloudify-test-ca/ca.crt'
external_ca_cert_path: '/root/.cloudify-test-ca/ca.crt'
prometheus:
credentials:
username: 'prometheususername'
password: 'strongprometheuspasswordnode2'
cert_path: '/root/.cloudify-test-ca/hostname2.example.com.crt'
key_path: '/root/.cloudify-test-ca/hostname2.example.com.key'
ca_path: '/root/.cloudify-test-ca/ca.crt'
services_to_install:
- manager_service
- monitoring_service
- entropy_service
cfy_manager install -c /etc/cloudify/manager_config.yaml
Example manager_config.yaml for Node 3:
# /etc/cloudify/manager_config.yaml
manager:
private_ip: '192.0.2.3'
public_ip: '203.0.113.3'
security:
ssl_enabled: true
admin_password: 'strongadminpasswordnode3'
cloudify_license_path: '/root/cloudify/license.yaml'
postgresql_server:
ca_path: '/root/.cloudify-test-ca/ca.crt'
cluster:
nodes:
hostname1:
ip: '192.0.2.1'
hostname2:
ip: '192.0.2.2'
hostname3:
ip: '192.0.2.3'
postgresql_client:
server_password: 'strongserverpassword'
ssl_enabled: true
networks:
default: '192.0.2.3'
rabbitmq:
username: 'rabbitmqusername'
password: 'strongrabbitmqpassword'
ca_path: '/root/.cloudify-test-ca/ca.crt'
cluster_members:
hostname1:
networks:
default: '192.0.2.1'
hostname2:
networks:
default: '192.0.2.2'
hostname3:
networks:
default: '192.0.2.3'
ssl_inputs:
external_cert_path: '/root/.cloudify-test-ca/hostname3.example.com.crt'
external_key_path: '/root/.cloudify-test-ca/hostname3.example.com.key'
internal_cert_path: '/root/.cloudify-test-ca/hostname3.example.com.crt'
internal_key_path: '/root/.cloudify-test-ca/hostname3.example.com.key'
ca_cert_path: '/root/.cloudify-test-ca/ca.crt'
external_ca_cert_path: '/root/.cloudify-test-ca/ca.crt'
prometheus:
credentials:
username: 'prometheususername'
password: 'strongprometheuspasswordnode3'
cert_path: '/root/.cloudify-test-ca/hostname3.example.com.crt'
key_path: '/root/.cloudify-test-ca/hostname3.example.com.key'
ca_path: '/root/.cloudify-test-ca/ca.crt'
services_to_install:
- manager_service
- monitoring_service
- entropy_service
cfy_manager install -c /etc/cloudify/manager_config.yaml
Post Installation
Once the database, broker, and manager are installed, perform the following on the manager node.
cfy cluster db-nodes list
cfy cluster brokers list
cfy cluster managers list
cfy cluster status