Subcloud Backup
Use the Subcloud Backup feature to customize backup configurations and organize subclouds across deployments, allowing for automated, scheduled subcloud backups with ease.
Backup Policy Creation:
Configure detailed settings for subcloud backup groups.Each policy can be reused and shared across multiple backup groups, ensuring flexibility and consistency.
| Options | Definition |
|---|---|
| Name | The unique name of the policy |
| Maximum Parallel Backups | Within each workflow, the maximum number of subcloud backup can be executed at the same time |
| Maximum Retires | For a failed subcloud backup operation, the number of retries before claim operation fails |
Backup group Creation:
Use Backup Groups to organize subclouds using selectors, applying a tailored backup policy to initiate backup operations instantly or according to a recurring schedule.
| Options | Definition |
|---|---|
| Name | The unique name of the backup group |
| System Selectors* | Labels that to be used to group matching subclouds Selectors |
| Now | Start a workflow now for subclouds that match system selectors |
| Execution Date | Choose start date and end date (optional) for scheduled workflow |
| Recurrence | Input the recurrence interval for rescheduling the workflow, i.e. every 1 hour |
*System Selectors only support unique selector keys; if an existing key is used, the new value will overwrite the previous one, rather than allowing multiple keys with the same name.
Backup Group table:
| Options | Definition |
|---|---|
| State | State of a backup group: enabled or disabled. if a backup group is without schedule, when it is in enabled state, a workflow is in progress or queued up if a backup is with schedule, when it is in enabled state, workflow has been scheduled on conductor and will start workflow based on recurrence |
| Name | Name of the backup group |
| Backup Policy ID | Name of the backup policy that is being used for this backup group |
| Execution Schedule | Start date, End date (optional) and Recurrence of this backup group |
| System Selectors | The list of all the selectors have been used in this backup group |
| Execution Status | Status for each workflow, details can be found in workflow status table |
| Run Now | Initiate an immediate subcloud backup workflow for the backup group |
| Update Group | Edit group current configurations |
| Delete Group | Delete the backup group |
| List Subclouds | List to show subclouds that belong to the group |
List subclouds window:
Workflow Status:
| Operation Name | Description |
|---|---|
| workflow_succeeded | workflow completed successfully |
| workflow_queued | workflow queued, conductor is busy with other workflow. workflow will start once in progress workflows complete |
| workflow_failed | workflow failed |
| workflow_started | workflow started, operation is in progress |
| workflow_scheduled | workflow scheduled on conductor |
| workflow_schedule_failed | workflow scheduled failed on conductor |
Warning types:
| Warning contents | Description |
|---|---|
| NO subcloud is found | Based on the selector, there is no subcloud matches the backgroup |
| workflows failed to be deleted | On conductor, workflow failed to be deleted |
| workflows has issues | Workflow are either failed or cancelled |
| Schedule Expired | The backup group's schedule has expired; no further recurring subcloud backup operations will be initiated. |
| Subcloud Backup Failure | One or more subclouds' backup operation failed |
sysinv Backup related status:
| Status | Description |
|---|---|
| backup group created | subcloud belongs to a backup group |
| cancelled | subcloud backup operation has been cancelled |
| failed | subcloud backup operation failed |
| backup group disabled | backup group the subcloud belongs to has been disabled, no workflow is scheduled or in progress |
| in progress | backup operation is in progress |
| backup group scheduled | backup operation has been scheduled on conductor |
Color Legend:
Executions Status:
| Status | Color |
|---|---|
| workflow_scheduled/workflow_queued | grey |
| workflow_started/workflow_succeeded | green |
| others | red |
Subcloud List Status:
| Status | Icon Color |
|---|---|
| backup group disabled | grey |
| workflow_scheduled/workflow_queued | black |
| workflow_started/workflow_cancelled | orange |
| workflow_succeeded | green |
| workflow_failed/workflow_schedule_failed | red |
| others | grey |
Feature Notes:
How to start a subcloud backup workflow:
There are three ways to initiate a workflow for a backup group:
-
When create a backup group, choose now, this will initiate a one-time backup operation for the backup immediately.
-
When create a backup group, choose a schedule, this will schedule workflows target on matching subclouds, workflow will be scheduled according to the recurrence and the scheduled time range.
-
Choose "Run Now" for existing backup group, this will initiate subcloud backup workflow right away for matching subclouds.
Subcloud Duplicate Detection
A new backup group will not be created if it includes a subcloud that is already part of an existing backup group. This prevents a specific subcloud from having duplicate workflows for backup operations.
Max parallel backup Strategy:
The Max Parallel Backup Strategy processes subcloud backups in manageable parallel batches, governed by key parameters like max_parallel_subclouds and max_retries. It begins by initializing the queue_pool with all subclouds and then iteratively processes them in batches, executing backups and monitoring their status. Failed backups are re-queued for retries up to the allowed limit, ensuring each subcloud has multiple attempts to succeed. The loop continues until all subclouds are either successfully backed up or have failed after exhausting retries, with detailed logging throughout to track progress and issues.
Backup Warning Message:
Updating a backup group will reset and recreate warning messages as necessary. If the warning message ‘workflows have issues’ is displayed, it will remain until the next workflow execution is completed and the status is updated with a newer execution status.
Backup Group Workflow Schedule Strategy:
Based on Backup group selectors, WRC identifies matching subclouds and creates workflows on the conductor for each deployment to which those subclouds belong.
Schedule workflows:
A scheduled workflow is a recurring workflow with a configured recurrence pattern. When it’s time to execute, the scheduled workflow triggers the creation of a workflow. Disabling an enabled backup group with a schedule will remove the associated scheduled workflow from the conductor.
New Subcloud matching Strategy
If a new subcloud is created that matches multiple backup groups, it will be assigned to the most appropriate backup group based on the following criteria:
-
Most Label Matches: The subcloud will be placed in the backup group with the highest number of matching labels.
-
Deployment Consideration: If multiple groups have the same number of label matches, the subcloud will be assigned to the backup group already has workflow created to the deployment, with priority given to the first group found on the list, sorted alphabetically.
Initiate workflow immediately for “Run now” type of workflow after update selectors
After update the selector, please disable the backup group and then choose “run Now” to initiate the workflow immediately.
Password Change:
When sysadmin password is changed for system controller manually (due to expiry or some other reason)
-
Ensure the related k8s secret is updated accordingly in your Kubernetes cluster.
-
Ensure that new sysadmin password is synced to the subclouds by ssh to the subclouds with new password. If not synced automatically then subclouds can be rebooted to get the new sysadmin password.
This sysadmin password consistency is essential for subcloud backup operation.
Workflow Status:
Scheduled Backup Group and Workflows
A scheduled backup group creates scheduled workflows, which are records indicating that a schedule has been set on the conductor. These schedules generate new workflows based on the provided timing. If there are active in-progress workflows, any newly initiated workflows will remain in a pending status until the previous workflows are completed.
Scheduled Type of Workflow
-
A workflow created by backup groups with a specified schedule that triggers workflows according to that schedule.
-
When a workflow is generated by a scheduled workflow, it is marked as a scheduled workflow before entering either the pending or in-progress status.
Workflow Status on the Conductor
The conductor only retains the status of the last completed workflow, whether it was successful or failed. The conductor’s interface will display the following:
-
Any scheduled workflows.
-
Any in-progress workflows.
-
The status of the last completed workflow.
Enabling a Disabled Backup Group
When you enable a previously disabled backup group, all prior workflow statuses are reset as workflows are re-scheduled on the conductor. If a backup group is disabled and then re-enabled while there are in-progress workflows, the status of those workflows will be lost.