Suspend Plugin

These features are part of the utilities plugin.

Additional support for suspend, resume, backup, restore, remove_backup workflows.

The plugin supports

suspend: Workflow call cloudify.interfaces.freeze.suspend for each node that has such operation. For backward compatibility also run deprecated. cloudify.interfaces.lifecycle.suspend.
resume: Workflow call cloudify.interfaces.freeze.resume for each node that has such operation. For backward compatibility also run deprecated. cloudify.interfaces.lifecycle.resume.
backup: Workflow call such calls for each node that has such operation.
- cloudify.interfaces.freeze.fs_prepare for all services nodes,
- cloudify.interfaces.freeze.fs_prepare for all compute nodes,
- cloudify.interfaces.snapshot.create for all nodes in deployment,
- cloudify.interfaces.freeze.fs_finalize for all compute nodes,
- cloudify.interfaces.freeze.fs_finalize for all services nodes.
restore: Workflow call such calls for each node that have such operation.
- cloudify.interfaces.freeze.fs_prepare for all services nodes,
- cloudify.interfaces.freeze.fs_prepare for all compute nodes,
- cloudify.interfaces.snapshot.apply for all nodes in deployment,
- cloudify.interfaces.freeze.fs_finalize for all compute nodes,
- cloudify.interfaces.freeze.fs_finalize for all services nodes.
remove_backup: Workflow call cloudify.interfaces.snapshot.delete for each node that has such operation.
statistics: Workflow call cloudify.interfaces.statistics.perfomance for each node that has such operation.

Suspend/Resume support by plugins:

Plugin	VM Suspend/Resume	File System freeze/unfreeze
Openstack 2.9.0+	Y	N (N/A)
vSphere 2.7.0+	Y	N (N/A)
vSphere 3.0.0+	Y	N (N/A)
LibVirt 0.4+	Y	N (N/A)

Backup/Snapshot support by plugins:

Plugin	VM Snapshot	VM Backup	VM Snapshot Restore	VM Backup Restore	VM Snapshot Remove	VM Backup Remove
Openstack 2.9.0+	Y	Y	Y	Y	Y	Y
Openstack 3.0.0+	Y	Y	Y	Y	Y	Y
vSphere 2.7.0+	Y	N (No API)	Y	N (No API)	Y	N (No API)
LibVirt 0.6.0+	Y	Y	Y	Y (Bypassed)	Y	Y

Plugin	Network Snapshot	Network Backup	Network Snapshot Restore	Network Backup Restore	Network Snapshot Remove	Network Backup Remove
LibVirt 0.4.1+	Y	Y	Y (By Compare)	Y (By Compare)	Y	Y

Plugin	Volume Snapshot	Volume Backup	Volume Snapshot Restore	Volume Backup Restore	Volume Snapshot Remove	Volume Backup Remove
Openstack 2.9.0+	Y	Y	N (No API)	Y	Y	Y
Openstack 3.0.0+	Y	Y	N (No API)	Y	Y	Y
vSphere 2.7.0+	N (N/A)	N (N/A)	N (N/A)	N (N/A)	N (N/A)	N (N/A)
LibVirt 0.4.1+	Y (By VM)	Y (Bypassed)	Y (By VM)	N (No API)	Y (By VM)	Y (Bypassed)
GCP 1.5.0+	N (N/A)	Y	N (N/A)	N (N/A)	N (N/A)	Y

Notes:

Abbreviations:

N/A - Not supported by plugin.
Y - Supported by plugin.
Y (Bypassed) - Can be bypassed by separate API or property.
Y (By VM) - Created automatically by run action on connected VM.
Y (By Compare) - Code show difference between states without reconfigure object.
N (No API) - Unsupported by infrastructure API.

All workflows have support for:

include_instances parameter for specifying list of instances id’s that operations will take place on them.
skip_actions node property check for disabling specific actions on the node.

Backup types:

Snapshot has such meaning - some objects that directly connected to parent object (like: VM/Compute/Volume) and what saved is difference between states of object.
Backup has such meaning - some objects contain a full copy of the original object and can be used after removing the original object.

For partial backup can be used include_instances for limit list of instances or split installation to several deployments and run on deployments one by one.

OpenStack:

Plugin will remove all binded snapshot before delete volume automatically.
Volume backups, VM backups, VM snapshots are removed only for delete backup workflow, in other cases user should remove images manually.
Plugins 2.7.+ and 3.0.+ can have different functionality coverage.

VSphere:

Plugin will remove all binded snapshot before delete vm.
User should remove snapshots before attach/detach devices from/to vm.

LibVirt:

Plugin will remove all binded snapshot before delete vm.
Plugin create xml backups in current directory by default.

Backup/Shapshot workflows:

We provide for use 3 workflows: create/restore/remove_backup.

Backup

The Backup workflow has the following parameters:

snapshot_name: Backup name/tag. By default will be used “backup-”
snapshot_incremental: Create incremental snapshots or full backup. By default created snapshots.
snapshot_type: The backup type, like ‘daily’ or ‘weekly’. By default: irregular
snapshot_rotation: How many backups to keep around. By default: 1

The meaning of each param depends on plugin implementation and can have a different sense for each plugin.

For example, openstack use parameters in such way:

Snapshot name: Used as suffix for created objects. As object can be different things like images, volume snapshots, or backups. The name of a resulted object is something like <object type>-<original object id>-<backup name>. We need that because the result of VM snapshot, VM backup, and Volume backup is an image, hence we need some information in the name to understand the id of the parent object for backup.
Snapshot_incremental: plugin use to separate type of resulted objects. If Snapshot_incremental=True - code will try to create snapshot of object, for different plugins it can be contain different information. In case Openstack it will be Image with full copy of VM or subobject contained in volume that can be exported as image. If Snapshot_incremental=False - code will try to create copy of VM as image or copy of Volume as image.
snapshot_rotation: used only with VM snapshot as rotation field in metadata.
snapshot_type: used with VM snapshots as part of image metadata and as description in snapshot metadata.

OpenStack plugin does not make any decisions based on snapshot_type / snapshot_rotation values and is passing these values without any changes. The plugin is not responsible for removing all old backups or snapshots by rotation field.

Restore

The Restore workflow has the following parameters:

snapshot_name: Backup name/tag. By default will be used “backup-”
snapshot_incremental: Restore from incremental snapshots or full backup. By default restored from snapshots.

Both parameters have the same meaning as in backup workflow.

For openstack:

VM’s: Code search images with same name as we used for create backup/snapshot image and rebuild VM with use such name as base.
Volumes: if the customer has tried to restore from a snapshot - we show a warning and ignore the action. In case the customer has used backup - we ask OpenStack to restore the volume from backup. Such logic is a limitation of OpenStack, hence we can only restore volumes from backups for now.

Remove backup

The Remove backup workflow has the following parameters:

snapshot_name: Backup name/tag. By default will be used “backup-”
snapshot_incremental: Delete incremental snapshots or full backup. By default removed snapshots.

Both parameters have the same meaning as in backup workflow.

For OpenStack:

VM’s - search image created by backup workflow and delete if found such.
Volume - search image created as backup for volume or remove snapshot with such name in volume.

Internal implementation/logic in utilities plugin.

In backup/restore workflow:

call action cloudify.interfaces.freeze.fs_prepare for all non compute nodes. This action is supposed to stop service before run real backup.
call action cloudify.interfaces.freeze.fs_prepare for compute nodes. This action is supposed for run sync/freeze fs before run real backup.
call action cloudify.interfaces.snapshot.create for all nodes in deployment. This action is supposed for low level create backup of volume or vm in infrastructure. if user implemented callback for action - can be used for made backup of service db.
call action cloudify.interfaces.freeze.fs_finalize for compute nodes. This action is supposed for run unfreeze fs before run real backup.
call action cloudify.interfaces.freeze.fs_finalize for all non compute nodes. This action supposed for start all serviced stopped at the start of workflow.

Plugin needs to run fs_prepere for restore workflow - because plugin needs to have consistent state before run any restore/apply backups - so if service has some suspend before backup we need to stop candidate for restore before any real action.

All actions receive all params as we have as inputs for workflow without any changes. So user defined actions can make some decisions based on such params.

In remove backup workflow plugin calls remove backup action on all instances in deployment without any additional actions for freeze or stop services. So remove backup is safe action without functionality degradation of deployment.

Usage example:

1. Upload the blueprint :

cfy blueprints upload example.yaml

2. Create deployment:

cfy deployments create -b cloudify-suspend

3. Call ‘suspend’ workflow:

Suspend:

$ cfy execution start suspend -d cloudify-suspend 
Executing workflow `suspend` on deployment `cloudify-suspend` [timeout=900 seconds]
2020-05-27 14:06:39.552  CFY <cloudify-suspend> Starting 'suspend' workflow execution
2020-05-27 14:06:39.556  CFY <cloudify-suspend> [server_0chf3l] Starting to cloudify.interfaces.lifecycle.suspend
2020-05-27 14:06:39.821  CFY <cloudify-suspend> [server_0chf3l.suspend] Sending task 'script_runner.tasks.run'
2020-05-27 14:06:41.641  LOG <cloudify-suspend> [server_0chf3l.suspend] INFO: Downloaded scripts/suspend.py to /tmp/3VLF5/suspend.py
2020-05-27 14:06:41.701  LOG <cloudify-suspend> [server_0chf3l.suspend] INFO: suspend server_id=Server!
2020-05-27 14:06:42.420  CFY <cloudify-suspend> [server_0chf3l.suspend] Task succeeded 'script_runner.tasks.run'
2020-05-27 14:06:42.523  CFY <cloudify-suspend> [server_0chf3l] Done cloudify.interfaces.lifecycle.suspend
2020-05-27 14:06:42.692  CFY <cloudify-suspend> 'suspend' workflow execution succeeded

4. call resume workflow:

$ cfy execution start resume -d cloudify-suspend 
Executing workflow `resume` on deployment `cloudify-suspend` [timeout=900 seconds]
2020-05-27 14:06:08.910  CFY <cloudify-suspend> Starting 'resume' workflow execution
2020-05-27 14:06:08.915  CFY <cloudify-suspend> [server_0chf3l] Starting to cloudify.interfaces.lifecycle.resume
2020-05-27 14:06:09.179  CFY <cloudify-suspend> [server_0chf3l.resume] Sending task 'script_runner.tasks.run'
2020-05-27 14:06:10.970  LOG <cloudify-suspend> [server_0chf3l.resume] INFO: Downloaded scripts/resume.py to /tmp/TXFUE/resume.py
2020-05-27 14:06:11.044  LOG <cloudify-suspend> [server_0chf3l.resume] INFO: resume server_id=Server!
2020-05-27 14:06:11.773  CFY <cloudify-suspend> [server_0chf3l.resume] Task succeeded 'script_runner.tasks.run'
2020-05-27 14:06:11.787  CFY <cloudify-suspend> [server_0chf3l] Done cloudify.interfaces.lifecycle.resume
2020-05-27 14:06:11.957  CFY <cloudify-suspend> 'resume' workflow execution succeeded

More executions logs:

Create backup:

$ cfy executions start backup -b examples -p snapshot_name=backup_example --task-retry-interval 30 --task-retries 30
2018-05-16 12:10:22.408  CFY <examples> Starting 'backup' workflow execution
2018-05-16 12:10:22.413  CFY <examples> [example_node_s4bgna] Starting to cloudify.interfaces.freeze.fs_prepare
2018-05-16 12:10:22.413  CFY <examples> [qemu_vm_jvv6jt] Starting to cloudify.interfaces.snapshot.create
2018-05-16 12:10:22.413  CFY <examples> [example_node_s4bgna] Starting to cloudify.interfaces.freeze.fs_finalize
2018-05-16 12:10:22.512  CFY <examples> [example_node_s4bgna.fs_finalize] Sending task 'cloudify_terminal.tasks.run'
2018-05-16 12:10:22.512  CFY <examples> [qemu_vm_jvv6jt.create] Sending task 'cloudify_libvirt.domain_tasks.snapshot_create'
...
2018-05-16 12:10:47.604  CFY <examples> [example_node_s4bgna] Done cloudify.interfaces.freeze.fs_finalize
2018-05-16 12:10:47.604  CFY <examples> [qemu_vm_jvv6jt] Done cloudify.interfaces.snapshot.create
2018-05-16 12:10:47.681  CFY <examples> [example_node_s4bgna] Done cloudify.interfaces.freeze.fs_prepare
2018-05-16 12:10:47.767  LOG <examples> INFO: Backuped to u'backup_example'
2018-05-16 12:10:47.768  CFY <examples> 'backup' workflow execution succeeded

Restore backup:

$ cfy executions start restore -b examples -p snapshot_name=backup_example --task-retry-interval 30 --task-retries 30
2018-05-16 12:12:43.913  CFY <examples> Starting 'restore' workflow execution
2018-05-16 12:12:43.917  CFY <examples> [example_node_s4bgna] Starting to cloudify.interfaces.freeze.fs_finalize
2018-05-16 12:12:43.917  CFY <examples> [qemu_vm_jvv6jt] Starting to cloudify.interfaces.snapshot.apply
2018-05-16 12:12:43.917  CFY <examples> [example_node_s4bgna] Starting to cloudify.interfaces.freeze.fs_prepare
...
2018-05-16 12:13:13.114  CFY <examples> [example_node_s4bgna] Done cloudify.interfaces.freeze.fs_prepare
2018-05-16 12:13:13.229  CFY <examples> [example_node_s4bgna] Done cloudify.interfaces.freeze.fs_finalize
2018-05-16 12:13:13.314  LOG <examples> INFO: Restored from u'backup_example'
2018-05-16 12:13:13.314  CFY <examples> 'restore' workflow execution succeeded

Delete backup:

$ cfy executions start remove_backup -b examples -p snapshot_name=backup_example --task-retry-interval 30 --task-retries 30
2018-05-16 12:14:42.171  CFY <examples> Starting 'remove_backup' workflow execution
2018-05-16 12:14:42.174  CFY <examples> [qemu_vm_jvv6jt] Starting to cloudify.interfaces.snapshot.delete
2018-05-16 12:14:42.275  CFY <examples> [qemu_vm_jvv6jt.delete] Sending task 'cloudify_libvirt.domain_tasks.snapshot_delete'
2018-05-16 12:14:42.322  CFY <examples> [qemu_vm_jvv6jt.delete] Task started 'cloudify_libvirt.domain_tasks.snapshot_delete'
2018-05-16 12:14:42.364  LOG <examples> [qemu_vm_jvv6jt.delete] INFO: remove_backup
2018-05-16 12:14:42.429  LOG <examples> [qemu_vm_jvv6jt.delete] INFO: Backup deleted: vm-backup_example
2018-05-16 12:14:42.430  CFY <examples> [qemu_vm_jvv6jt.delete] Task succeeded 'cloudify_libvirt.domain_tasks.snapshot_delete'
2018-05-16 12:14:42.499  CFY <examples> [qemu_vm_jvv6jt] Done cloudify.interfaces.snapshot.delete
2018-05-16 12:14:42.578  LOG <examples> INFO: Removed u'backup_example'
2018-05-16 12:14:42.578  CFY <examples> 'remove_backup' workflow execution succeeded