Upgrade Operator and Entity Model - Operations at Scale (OAS)

Introduction

Wind River Conductor leverages Kubernetes operators to manage platform upgrades (WRCP). The model works in a declarative way, in which the user is required to provide the desired target state via the Kubernetes API. These process is also referred to as Operations at Scale (OAS).

The figure below illustrates how the entities interact with one another to setup and trigger upgrades.

Installation

To install OAS see Installing Containerized Wind River Conductor (HELM).

Note:

In order for the full “Operations at Scale” Conductor solution to work, the following prerequisites are required:

• WRCP host with a Kubernetes cluster to install Conductor
• A functional containerized instance of WRC running and reachable by OAS Pods
• WRCP blueprint called wrcp-blueprint and WRCP plugin uploaded to Conductor

Check OAS installation

To check if OAS pods was successfully deployed you can:

• Check pods - OAS is correctly running if System Inventory Manager, Upgrade Group Manager and WRC Secret Operator are up:

kubectl get pods -n <namespace>

NAME                                        READY   STATUS    RESTARTS        AGE
api-service-666488ddc8-stftx                1/1     Running   0               3d
composer-backend-57bd8f8989-cl6gq           1/1     Running   0               3d
composer-frontend-5687df978c-v8pkh          1/1     Running   0               3d
execution-scheduler-64b977cfc-fkwc2         1/1     Running   0               3d
kube-state-metrics-84df7548f9-4xm6t         1/1     Running   0               3d
mgmtworker-0                                1/1     Running   4 (2d23h ago)   3d
nginx-6fd7bbb57f-zclq2                      1/1     Running   0               3d
postgresql-0                                2/2     Running   0               3d
prometheus-server-778b7ff955-w5f64          1/1     Running   0               3d
rabbitmq-0                                  1/1     Running   0               3d
rest-api-server-b85b5f756-dl9pf             1/1     Running   0               3d
rest-service-64f895db75-rm6jl               1/1     Running   0               3d
seaweedfs-filer-0                           1/1     Running   0               3d
seaweedfs-master-0                          1/1     Running   0               3d
seaweedfs-s3-579546544d-f5rkp               1/1     Running   0               3d
seaweedfs-volume-0                          1/1     Running   0               3d
stage-backend-64c844f4cc-n2jz7              1/1     Running   0               3d
stage-frontend-554fbbbb56-fhl2h             1/1     Running   0               3d
system-inventory-manager-675dc69cd9-kztdx   1/1     Running   0               2d23h
upgrade-group-manager-bf7fd6965-c5zf5       1/1     Running   0               20m
wrc-secret-operator-f9b5b5cb5-tmjr8         1/1     Running   0               3d

• Check CRDS - The CRDs available will be System Inventory, Upgrade Group and Upgrade Policy.

kubectl get crds -n <namespace>

NAME                                   CREATED AT
systeminventories.wrcp.windriver.com   2024-03-15T13:14:59Z
upgradegroups.wrcp.windriver.com       2024-03-15T13:14:59Z
upgradepolicies.wrcp.windriver.com     2024-03-15T13:14:59Z

Note 01: If you change the username and password in the rest_service you must also change the same parameters inside the wrc_endpoint_secret in the values.yaml, the values must be in base64.

wrc_endpoint_secret:
  data:
    username: YWRtaW4=
    password: YWRtaW4=

Note 02: If you need a base 64 converter and don’t want a online tool you can use the python interpret. Record the value return in the single quotes.

>>> import base64
>>> base64.b64encode(b"admin")
b'YWRtaW4='

Definitions

System Inventory

• Represents an upgradable system and provides sufficient inventory to configure and report the current software versions and status.
• Manually created or automatically based for the discovered inventory.
• Configured directly or indirectly via Upgrade Operator.
• Individual and Component level version configuration and reporting to facilitate fine-grain control for feature activation.
• Status and progress is recorded via entity annotations to facilitate machine interaction.

Upgrade Group

• User defined request to upgrade a group of target systems – selected statically or dynamically.
• The specific systems are identified by the supplied system selector criteria.
• Upgrade groups can be permanent or transient, with systems belonging to multiple groups.
• Maintains overall upgrade progress and status for the upgrade group.
• Status and progress is recorded via entity annotations to facilitate machine interaction.

Upgrade Policy

• User defined control for the behavior and operations of the upgrade execution, including error handling and batching requirements.
• Policy is associated with the Upgrade Group entity to permit the sharing of the defined behavior.
• Execution Limits – Ability to specify operation timeout values for each workflow independently (precheck, upgrade, etc) or for specific tasks.
• Batch Size – Limit the number of WRC executions that can be performed in parallel for a specific workflow or restrictions for a given set of systems.

System Inventory Manager

The System Inventory Manager is responsible for:

• Initiating sub-cloud discovery for System Inventory entities with System Controller role.
• Discovering systems created as new system inventory entities.
• Periodically querying clusters via the WRCP API for current properties, component versions and system state.
• Creating and updating system inventory entities based on inventory discovery and synchronization.
• Creating and updating conductor secrets for system access.
• Creating deployments for newly configured or discovery systems.

Upgrade Group Manager

The Upgrade Group Manager is responsible for:

• Starting the schedule process.
• Checking differences in the Upgrade Group Object to trigger the specific workflow for it.
• Undo de schedules once the Upgrade Group Object is deleted.
• Keep track of the related System Inventory Objects.

Secret Operator

The Secret Operator is responsible for:

• Allowing the connection of System Inventory Manager with WRC and WRCP.
• Storing the credentials to connect Upgrade Group Operator ans System Inventory Manager with WRC.
• Replicating WRCP Secrets to WRC.

Kubernetes Events

The Kubernetes Publisher is responsible for:

• Creating new Kubernetes Events with the information contained in the WRC workflow messages.

System Overview

The user creates System Inventory, Upgrade Group and Upgrade Policy CRDs and inputs them into Cluster through kubectl command. Depending on the System Inventory type, the System Inventory Manager discovers sub-clouds and creates new System Inventories and deployments in WRC. The Upgrade Group Operator notices the existence of those new System Inventories and may trigger or schedule the WRC workflow to upgrade WRCP hosts to new versions. The Upgrade Group Operator receives workflow status from WRC and update System Inventories and Upgrade Groups properly.

The System Inventory Manager frequently gets data from WRCP hosts to keep the inventory up-to-date. If discrepancies between the original values in the System Inventory and the current ones in WRCP hosts are identified, the System Inventory Manager reports them to user. It is possible to reconcile the discrepancies by using the WRC user interface.

The Upgrade Group Manager monitors changes in the System Inventories and acts accordingly to update Upgrade Groups.

All actions performed via kubectl can be performed using WRC user interface.

Access Control

• Access and authorization is controlled using standard Kubernetes authentication and RBAC policies which is enforced via the Kubernetes API.
  • CRDs that define the Upgrade and System Inventory entities can use RBAC rules to control admin, edit and view privileges for individual resources or operations.
  • Resource access controls may also be defined for namespaces to facilitate organizational requirements.
• System-level access controls will be enforced by the authorization granted by the credentials provided to access and manage the configured systems.
  • System access information is maintained in Secrets which is referenced by the System Inventory entity and may be shared across systems within that namespace.
  • Discovered system access credentials will be populated with Secrets in the same namespace as the system that initiated the discovery (i.e. System Controller).

Clean-up Troubleshooting

In order to preform a clean up when uninstalling, the Custom Resources (System Inventories, Upgrade Groups, and Upgrade Policy) must be deleted prior to uninstalling helm. If not, some resources might get frozen when trying to delete it. This happens because the operator will be down at the moment of deletion, which is a limitation from kopf.

If helm is uninstalled before the resources are deleted, the kopf finalizer must be manually removed using the following command:

kubectl patch <resource_type> <resource_name> -p '{"metadata": {"finalizers": []}}' --type merge -n <namespace>

The resource_type can be: SystemInventory, UpgradeGroup, UpgradePolicy or Secret.

NOTE: Do NOT remove the finalizer from the CRD, since it can leave orphaned custom resource data in etcd.

Glossary

• Deployment ID: WRC deployment ID
• Deployment: WRC deployment
• Discovery: The discovery of the WRCP hosts
• Execution: WRC workflow execution
• Operator: Kubernetes operator
• plcy, policy: Short name of Upgrade Policy object
• RBAC: Kubernetes Role-based access control
• Secret: Kubernetes secret
• sysinv: Short name of System Inventory object
• Upgrade: WRC Upgrade workflow
• upgrp: Short name of Upgrade Group object
• Workflow: WRC workflow
• WRCP Central Cloud: Distributed Cloud system consists of a Central Cloud and one or more sub-clouds connected to the Central Cloud over L3 networks
• WRCP Standalone host: Any WRCP host that is not a Central Cloud or a sub-cloud
• WRCP Sub-cloud: The sub-clouds are WRCP instances used to host containerized applications

Command Line

To interact with CRDs it is necessary to use the kubectl command. All commands assume that the given CRD and Secret YAML files are already created by the user.

CRDs

OAS Custom Resource Definitions

There are 3 new CRDs that need to be managed:

1. System Inventories, short name is sysinv.
2. Upgrade Groups, short name is upgrp.
3. Upgrade Policy, short name is upgradepolicy.

You can interact with these resources using the standard Kubernetes CLI or the System Invetory, Upgrade Group and Upgrade Policy widgets.

Secret

A WRC Secret is defined in the following manner:

An example of the script to instantiate a WRC Secret is shown below:

apiVersion: v1
kind: Secret
metadata:
  name: wrc-endpoint-secret
  namespace: wrcp
  labels:
    app: conductor
data:
  USERNAME: username
  PASSWORD: password
stringData:
  HOST_URL: http://localhost:80/
  TRUST_ALL: "true"
  TENANT: "default_tenant"
  API_VERSION: "v3.1"
type: Opaque

A WRCP Secret is defined in the following manner:

An example of the script to instantiate a WRCP Secret is shown below:

apiVersion: v1
kind: Secret
metadata:
  name: wrcp-endpoint-my-controller
  namespace: wrcp
  labels:
    app: conductor
  annotations:
    wrcp-secret: "yes"
    wrcp-status: Ready
data:
  OS_USERNAME: username
  OS_PASSWORD: password
  SSH_USERNAME: sshusername
  SSH_PASSWORD: sshpassword
   CACERT: cacertdata
stringData:
  INSECURE: "false"
  OS_AUTH_URL: http://1.1.1.1:5000/v3
  OS_AUTH_TYPE: password
  OS_IDENTITY_API_VERSION: "3"
  OS_INTERFACE: internal
  OS_KEYSTONE_REGION_NAME: RegionOne
  OS_PROJECT_DOMAIN_NAME: Default
  OS_USER_DOMAIN_NAME: Default
  OS_PROJECT_NAME: admin
  OS_REGION_NAME: RegionOne
type: Opaque

An example of the script to instantiate a WRCP Secret with CACERT: Certificate encode in base64 is shown below:

apiVersion: v1
kind: Secret
metadata:
  name: wrcp-endpoint-dc
  namespace: wrcp
  labels:
    app: conductor
data:
  OS_USERNAME: YWRtaW4=
  OS_PASSWORD: TGk2OW51eCo=
  SSH_USERNAME: c3lzYWRtaW4=
  SSH_PASSWORD: TGk2OW51eCo=
  CACERT: |
    LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUZwVENDQTQyZ0F3SUJBZ0lVUmd6dGg1MXg0
    TnZ0ZEZ1L1EzSFkvcjdESnZNd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lqRUxNQWtHQTFVRUJoTUNW
    MHN4RURBT0JnTlZCQWdNQjFkaGEyRnVaR0V4RURBT0JnTlZCQWNNQjFkaAphMkZ1WkdFeEN6QUpC
    Z05WQkFvTUFsZFNNUXd3Q2dZRFZRUUxEQU5QY21jeEZEQVNCZ05WQkFNTUMzZGhhMkZ1ClpHRXVZ
    Mjl0TUI0WERUSTBNRFF3T1RFNU1qWTBORm9YRFRNME1EUXdOekU1TWpZME5Gb3dZakVMTUFrR0Ex
    VUUKQmhNQ1Ywc3hFREFPQmdOVkJBZ01CMWRoYTJGdVpHRXhFREFPQmdOVkJBY01CMWRoYTJGdVpH
    RXhDekFKQmdOVgpCQW9NQWxkU01Rd3dDZ1lEVlFRTERBTlBjbWN4RkRBU0JnTlZCQU1NQzNkaGEy
    RnVaR0V1WTI5dE1JSUNJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQWc4QU1JSUNDZ0tDQWdFQXlH
    ZHlmWHhwQzVNSzM5V0JWbU9kdXgwSVpuY3AKdUlvd0Q4WXMwcFA3TmZRWnpNUjhXNTJiTExPb2RX
    UEduemVyTVhYTEtINjRBdlZQMDQ1SmNobDNOSTRKVnVJawo3U1NEL05PL1ZXYUkwSlJNREYwdmc5
    VlBxa080VXN1OGZzcFhzQThTbmFNUWJkMUFiekdzcWxFdTU3eUZVOGZWCkVFekVLM0xxc09qSWhp
    cVpPekF2c2t6RG91ZFFpaU8zem9uYWZLd1lxV3BTK043aDlXR041NmZ1ZmtkK2UvWlQKcFVYS21D
    VHNtaFpqNEFvdnhDN2xHVCt6TDY2NnUxY01hTkZyWFNwTXFhWlhMcFpTV3grVUZaWEVlYnFYaGZW
    RApkWGVNMUZHQy9tbStmRWdQZ3ExSmpwQjdGdWhYQmNLbmhnSDRoekZ5cTZkbDcxdlRxanJMU2Zm
    TXJjb3FWb3NLCnp0dVIvVjRNbHFjN1BNNVVvY2Y0WEw5dHhYU3NXTXg3WmtLQlpVOTlKTXFsdHJY
    aHV5ZXlzblNDallEYTV2TG8KZVk1bGtSU1JXU0dEckoyU3JTYkQrajF1TklwRjI1eWpILzdxZ0Rh
    MWVxQ2Jjc05xNDMvVDZRa2grTWZDQWY0MQpNdFl5N3p0UHcrUXZWblppWWh2aWhCbG9YVE1qcFdn
    WGxraHljWFRJVnVYZnkyMlFPUnprdEJyejhSdkNVRHZDCkRCKzR0aXpYMTU3S3hNVjlhM3dyNDdz
    TmYybmsraEt1V1ErZVlNREVENmkvc1F2YUxBbEN3a29tZ3UxS2FyZlQKUC9wLzhucEFGdGxJUzhY
    NXhlWGlUMFVuUERQb2d2UWxEcmFLMElJV2hhT2N3SGdaeXdpbHNwcHg0U0NnUHBsdApsVW1oSjFB
    U2xLU0NQejBDQXdFQUFhTlRNRkV3SFFZRFZSME9CQllFRkFvTVNoTTQrZ2xKOHRPQzhBU3lHZURO
    CnMxYzNNQjhHQTFVZEl3UVlNQmFBRkFvTVNoTTQrZ2xKOHRPQzhBU3lHZUROczFjM01BOEdBMVVk
    RXdFQi93UUYKTUFNQkFmOHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnSUJBSmIvd0hqK0FIc0NiU0ZL
    Sk83UDFXTGp4Sng0ZU5uRgphRlBFM2ZRcFY2RksvLzVBQ3R2aDlOdzN3Z244eE5ZbG1ramd4Tkxh
    elI1aStjd2t2YzNJWlZMaVA3Tmd1MWVuCnA2eTdVMTRvSFgzVW1FMTBPbmlDdFFWYXQzNjljTUs1
    WkljcE8rcW9scUFZOW00MHFFUDZ2blFORlJ0WnlXQk8KMEVobVBaenh2Vkw5TDViVWNzc1FTK0hV
    bmEzWnd4TTNDa1ZldUpiU2tPOTkxTXBJaXFNaFJ2SUNhMDJ1VVltdApyVjM2bVc5UFNqNVl3eGtP
    eGM4eXdyRmlhb0tXaUFYRTVXeHNBbENwMEhYQkk3eG1IVDV0MXBsS1dXYytaTzJ4CmxQTVNMNllV
    c1AyQjl4d0pJcWFncERPN0lwQU5nS2l5bVhxbTBQbDhBTnd3QXFUdStQWnFqRHpEQzVYSG8xUTkK
    Q3Y2eW9zZ29QNy9RMFJ4L0g0YW9VOXBpNWJYa0lOUWVIeERrSkdmSWFqd0ZETHBvSGt2VFZ4aXE0
    MnhZSUU1MQpXanhsTG9IWWVvSVRuendjS3d2STBuQzFCeEFhVXRZSzB0b21qbHBjT2JVejBVWHhP
    M2NrZnlwbzV3Rm9PVUphClJuTS9sNW9MUlQ0RVkzS0p2M3VDcFpwaytKYXIwczJxSnJrNzVMOENa
    Z2pLT1VrL1hUcmNZQk1MQ2NDaThOT3cKRmtKZkcraWd1cFN3Z3NYZThrS1NObkpKdFFhU1NOUmwy
    K1RkV3d5SE9ITXpITFVjYlJJOEh5aDVYZHFTTDdULwpjRHFBb2VmQ0IvOTNFUU5rRGUvMTNlL3Zi
    U2VxNjcwNEpXTm4xYThmSUZlQzBFVzlNdmNCRjJHRGdkNVdJZGJjCk9ObmtpazRrcEZnVAotLS0t
    LUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
stringData:
  INSECURE: "false"
  OS_AUTH_URL: http://1.1.1.1:5000/v3
  OS_AUTH_TYPE: password
  OS_IDENTITY_API_VERSION: "3"
  OS_INTERFACE: internal
  OS_KEYSTONE_REGION_NAME: RegionOne
  OS_PROJECT_DOMAIN_NAME: Default
  OS_USER_DOMAIN_NAME: Default
  OS_PROJECT_NAME: admin
  OS_REGION_NAME: RegionOne
type: Opaque

System Inventory

A System Inventory is defined in the following manner:

Ths following example demostrates how to instantiate a new System Inventory object:

apiVersion: wrcp.windriver.com/v1
kind: SystemInventory
metadata:
  name: my-controller
  namespace: wrcp
  annotations:
    upgrade-state: scheduled
    upgrade-stage: pending
    scheduled-upgrade-start: 2023-10-19T22:02:47Z
    scheduled-upgrade-stop: 2023-10-20T23:02:47Z
    deployment-upgrade: 48107489-82e9-426c-a8c1-63af98d7b11c
  labels:
    subcloud-group: region-foo
    release-version: "21.12"
    kubernetes-version: "1.23.1"
    distributed-cloud-role: subcloud
    system-controller: any-systemcontroller
    custom-label: custom-value
spec:
  wrcp-endpoint-secret: wrcp-endpoint-my-controller

Upgrade Group

The upgrade workflow requires ISO, License and Sig files to successfully upgrade WRCP hosts. These files must be placed in WRC mgmtworker Pod in the following paths:

• WRCP ISO: /opt/mgmtworker/persistent//default_iso.iso.
• WRCP License: /opt/mgmtworker/persistent//license.lic.
• WRCP Sig: /opt/mgmtworker/persistent//default_sig.sig.

Folder’s name must exactly match the name of Upgrade Group. The system will verify the path for each Upgrade Group before workflow execution.

An upgrade group is defined in the following manner:

The following example shows how to instantiate a Upgrade Group object:

apiVersion: wrcp.windriver.com/v1
kind: UpgradeGroup
metadata:
  name: testing-upgrade-group
  namespace: wrcp
spec:
  state: enabled
  policy: testing-policy
  selector:
    testing-label: testing-value
    release-version: "21.12"
  components:
    release-version: "22.12"
  schedule:
    upgrade:
      execution-date:
        start: "2100-07-30"
        end: "2100-08-30"
      execution-time:
        start: "10:30"
        end: "11:30"
      recurrence: "1d"
      count: 5

Upgrade Policy

Note: This CRD will be available in future releases. An upgrade policy is defined in the following manner:

The following example shows how to instantiate a Upgrade Policy object:

apiVersion: wrcp.windriver.com/v1
kind: UpgradePolicy
metadata:
  name: any
  namespace: wrcp
spec:
  batch-size:
    precheck: 5
  execution-limit:
    postcheck: 5
  max-retry:
    upgrade: 1
    prestaging: 1
  rollback-scope: system
  rollback-threshold :
    retry: 5
    abort: 1

Operators

Operators are responsible for the lifecycle of the management automation process for defined set of managed systems.

The Operator implements Kubernetes controllers for the system entities. The main purpose of any controller is to bring the actual state of the cluster to the desired state, as expressed with the resources/object specification.

Kubernetes Publisher

Architecture

The Kubernetes Publisher is composed of 2 main parts:

1. The listener: Responsible for reading the messages in the WRC AMQP queue.
2. The publisher: Responsible for translating the messages and “publishing” them as Kubernetes Events and System Inventory status updates.

Behavior

The Kubernetes Publisher listens to the WRC workflow messages coming from WRC RabbitMQ and does the following steps:

1. Checks whether the workflow in question is the upgrade workflow.
2. Checks whether the upgrade workflow has the upgrade strategy.
3. Looks for the corresponding System Inventory based on the deployment ID.
4. Creates a new Kubernetes Event with the information regarding the execution.
5. Updates the System Inventory status with its upgrade status.
   1. The possible statuses are: workflow_started, workflow_failed, workflow_cancelled and workflow_succeeded.

Checking the Events and the Status

The user can check whether the Kubernetes Publisher has created any events recently with the following command:

Events
kubectl get events

The user can also check the System Inventory status:

Status
kubectl describe systeminventories.wrcp.windriver.com deployment-name

Secrets operator

There are two types of Secrets to be considered: WRC and WRCP.

The WRC Secret is created by Helm installation and is used to establish communication between the System Inventory Manager and Upgrade Group Operator with the WRC.

WRCP Secrets contain the information needed to establish communication between System Inventory Manager with a WRCP hosts. The user can update WRC Secrets and create WRCP Secrets though kubectl. All WRCP Secrets are automatically propagated to WRC. Notice: The WRCP Secrets can be created just by CLI using the kubectl command.

WRCP Secret keys are propagated to WRC so they can be referenced by WRC deployments created by System Inventory Manager. The following table shows how Kubernetes Secret keys of a secret with name “wrcp-endpoint-my-controller” are translated to WRC Secrets:

The value of INSECURE is passed to the deployment as the ‘insecure’ input.

NOTE:

Only Kubernetes Secrets containing the keys listed in the table above are processed by the Secret Operator. Other secrets are processed once and ignored afterwards. When processing a non-WRCP secret, the Secret Operator raises an Exception so it can avoid trying to reprocess it. Unless the user is trying to create a WRCP Secret, this exception is expected:

[2024-02-15 11:50:21,341] kopf.objects         [ERROR   ] [wrcp/wrc-endpoint-secret-test] Handler 'handle_secret_creation' failed permanently: This secret does not contain expected keys. Don't try to process it again. Root cause: Failed to retrieve expected key 'OS_PASSWORD', skipping the creation of secrets in WRC.
[2024-02-15 11:50:21,341] kopf.objects         [INFO    ] [wrcp/wrc-endpoint-secret-test] Creation is processed: 0 succeeded; 1 failed.

System Inventory Manager

System Inventories can be created, updated, deleted, displayed and discovered through the Conductor widgets or Kubernetes CLI. The manual creation is allowed for System Inventories that represent a WRCP Central Cloud or a Standalone host. For WRCP sub-clouds its is not necessary any interaction of the user as they are discovered automatically by the system or on demand by the System Inventory widget.

Note: the release 24.03 is intended to allow the declarative interaction with Upgrade workflows that are only supported by Central Clouds, so the creation of non-Central Cloud System Inventories does not have effect. It is present just for compatibility with future releases.

When the System Inventory is changed it triggers process in Upgrade Group to keep all the data consistent. For example, when a new System Inventory is created the Upgrade Operator checks its labels. If the labels of the System Inventory match the selector of the Upgrade Group the System Inventory is impacted by this Upgrade Group, so Upgrade workflows may be scheduled to this System Inventory.

Discovery

The Discovery feature discover all the sub-clouds associated with a System Inventory that represents a Central Cloud. It is a periodic operation or can also be triggered manually by the user using the System Inventory widget. The action is available for System Inventories that represent Distributed Cloud only.

Note: in release 24.03 the periodicity is set as 4 hours.

Update host attributes

It is a periodic operation that get WRCP host data like component versions. It is the base operation to update discrepancies.

Note: in release 24.03 the periodicity is set as 4 hours.

Discrepancies

Discrepancies can happen when the information contained in a System Inventory object diverges from the information returned by the corresponding WRCP system. Ideally, the System Inventory object should always mirror the current state of the corresponding WRCP host, but issues may happen and they can cause this synchronization to be lost. Discrepancies can be audited using the Conductor widgets or the Kubernetes CLI and reconciled just via the Conductor widgets.

Upgrade Group Operator

Upgrade Groups can be created, updated, deleted, and displayed through the Conductor widgets or Kubernetes CLI.

Through the Upgrade Group selector the user can filter a set of System Inventories to be impacted by the WRC workflows. Depending on the value in “schedule” the workflow can be executed immediately or later.

Upgrade Group object creation

When an Upgrade group object is created, Upgrade Group Manager use the labels in selector field to filter System Inventories objects. Upgrade Group manager searches in System Inventories labels that match with selectors.

Note: The executions are created only for system controllers, subclouds will not have any schedule execution.

Draft system

It is possible to create an object as disable and any executions will be scheduled. Switching the state to enable, manager will filter System Inventories and create schedules.

Upgrade Group Updates

All executions are cancelled when something change in Upgrade Group. Manager’s filter System Inventories again and reschedule the executions.

Upgrade Group Deleted

When an object is deleted, all executions are cancelled.

Executions Summary

Upgrade Group objects store a summary of the executions inside the object in “spec.status” fields. Users can check how many System Inventories are scheduled, reconciled, in-progress and completed, further the total of System Inventories.

System Inventory Creation

When a System Inventory object is created, the Upgrade Group Manager checks if this new object belongs to an Upgrade Group object and if so, the manager schedules the executions.

System Inventory Updates

When a System Inventory object has its labels updated, the Upgrade Group Manager cancels the executions for the System Inventory and checks for an Upgrade Group match.

System Inventory Deleted

When a System Inventory is deleted, WRC cancels that deployment-id schedules.

Upgrade Controllers

To upgrade the controllers users must fill the Upgrade Group selector with the labels of desired controllers.

Upgrade and Prestage Subclouds

The Upgrade of subclouds in WRC 24.03 is run through the controllers, so users must set the filter to “controllers” in their group selectors. This will filter the controllers that will have their subclouds upgraded.

Controllers and Subclouds must have different Upgrade Groups and different schedules.

System inventories of subclouds will not have the status of the upgrade, since the upgrade run through the controllers, the status of subclouds upgrade will be shown in controller System Inventories.

• All subclouds: Set the parameter “subcloud-group” in selectors as “1”, in this case the filter of subclouds are made in WRCP and don’t need to previously create the group in platform.

• Group of subclouds

  1. Create the group of subclouds in WRCP and add desired subclouds to this group. In this specific case this are the filter for subclouds, so users don’t need to add “subcloud-group” label in System Inventories subclouds. If the group were created before the System Inventory, the group will be shown in object labels.
  2. Set the parameter “subcloud-group” in selectors with group name or id.

System Inventory - Widget

New Widgets

The cluster can be controlled using WRC’s web interface. To make this possible, some new widgets were introduced. Examples are shown below.

Note: Kubernetes works asynchronously, so in some cases it implies that users need to wait for a new status (refreshing page or waiting for a new data fetching, as defined in widgets configurations) or use kubectl describe resource to get more accurate data about resources.

System Inventory

All the System Inventories that have been created can be reviewed under Upgrades→System Inventory session:

Discovery

Run a discovery in WRCP to get recently added subclouds.

Audit System Inventories

Review any discrepancies between the object and the system, and address these issues through the User Interface (UI).

Add Inventory

Create new System Inventories by clicking on Add Inventory:

Filter System Inventories for Upgrade Group

List the objects that match System Inventory labels with Upgrade Group selector.

Upgrade Group - widget

Upgrade Group

The Upgrade Group list shows all upgrade groups created under Upgrade → System Upgrade:

To add a new Upgrade Group, click on Add Upgrade Group:

• Name: Object’s name. Not possible to rename.
• Executions:
  • Select desired workflow (upgrade or prestage).
  • Execution date - Date and time of start and end, can’t be a data in the past.
  • Count: How many times the workflow will run.
  • Recurrence: The frequency of executions, every day, every hour and so on.
  • Now: When set as True, will immediately run.
• Components Versions: The components available to upgrade. Only WRCP upgrade is available.
• System Selectors: Desired labels to filter System Inventories objects. Users can set a custom-label in System Inventory and type the key and value as a selector. WARNING:
  • System Inventories to be selected, MUST have all the labels selected.
  • Custom labels cannot mistype, otherwise Upgrade Group Manager will not select System Inventories correctly.
• State:
  • Draft: A draft Upgrade Group is just a placeholder, any execution will be schedule and if were switched to draft all executions are cancelled.
  • Active: An active Upgrade Group has all executions immediately scheduled, when created as or switched to.
• Upgrade Policy ID: All Upgrade Group policies available will be listed.
• Upgrade Options:
  • Force: Specifies whether to force the workflow execution in case there is already a running execution.

Upgrade Policy

Note: The Upgrade Policy is not used in release 24.03. It is there for future use.

Review Upgrade Policies that have been created under Upgrade → System Upgrade:

To add a new Upgrade Policy, click on Add Upgrade Policy:

• Name: Object’s name. Not possible to rename.
• Batch size: How many subclouds will run in parallel.
• Executions Limit(unavailable): The maximum time of running workflow.
• Max retry(unavailable): How many times a workflow can run after a failed execution.