Conductor Documentation

Google Cloud Plugin

The GCP plugin enables you to use Studio Conductor to manage Cloud resources on GCP. The currently supported resource types are listed later in this topic.

Note that some services and resources vary in availability between regions and accounts.

Plugin Requirements

Compatibility

The GCP plugin uses the official Google API Python Client.

Accessing Secrets

It is recommended that you store your credentials as secrets. You can do this using the CLI. Secrets can then be accessed inside your blueprints, as follows:

  network:
    type: cloudify.gcp.nodes.Network
    properties:
      gcp_config:
        auth:
          type: service_account
          auth_uri: https://accounts.google.com/o/oauth2/auth
          token_uri: https://accounts.google.com/o/oauth2/token
          auth_provider_x509_cert_url: https://www.googleapis.com/oauth2/v1/certs
          client_x509_cert_url: { get_secret: gcp_client_x509_cert_url }
          client_email: { get_secret: gcp_client_email }
          client_id: { get_secret: gcp_client_id }
          project_id: { get_secret: gcp_project_id }
          private_key_id: { get_secret: gcp_private_key_id }
          private_key: { get_secret: gcp_private_key }
        project: { get_secret: gcp_project_id }
        zone: { get_secret: gcp_zone }
      name: my_cloudify_network
      auto_subnets: false
 

GCP Plugin Configuration

The GCP plugin requires Service Account credentials and endpoint setup information in order to authenticate and interact with Google Cloud Provider. This is retrieved from your Google Cloud account.

To locate these credentials, nagivate to: APIs & Services::Credentials.

#!/bin/bash

export service_account_keys=/vagrant/service-account-keys.json

cfy secrets create gcp_client_x509_cert_url -u -s `cat ${service_account_keys} | jq -r '.client_x509_cert_url'`
cfy secrets create gcp_client_email -u -s `cat ${service_account_keys} | jq -r '.client_email'`
cfy secrets create gcp_client_id -u -s `cat ${service_account_keys} | jq -r '.client_id'`
cfy secrets create gcp_project_id -u -s `cat ${service_account_keys} | jq -r '.project_id'`
cfy secrets create gcp_private_key_id -u -s `cat ${service_account_keys} | jq -r '.private_key_id'`
cfy secrets create gcp_region -u -s us-east1
cfy secrets create gcp_zone -u -s us-east1b
cat ${service_account_keys} | jq -r '.private_key' | sed '$d' >> gcp-private-key
cfy secrets create gcp_private_key -u -f gcp-private-key
rm gcp-private-key

Google’s credential JSON file stores the private key as a string with \n string literals instead of line breaks. Before creating your gcp_private_key secret, the value needs to be transformed. Manually change this value, perform find and replace, or find another solution. Also, in the UI create secrets widget, do not use the text field store the key, rather upload the key as a file. CLI users can use the solution in the script above.

Another option to provide the credentials is to download the service account JSON key and store it and the gcp_zone as a secrets, as follows:

cfy secrets create gcp_credentials  -f <path_to_gcp_service_acoount_json>  
cfy secrets create gcp_zone -s <gcp_zone>

then your blueprint should look like that:

  network:
    type: cloudify.gcp.nodes.Network
    properties:
      gcp_config:
        auth: {get_secret: gcp_credentials}
        zone: { get_secret: gcp_zone }
      name: my_cloudify_network
      auto_subnets: false
 

Note:

cfy secrets create gcp_credentials -s <service_acocount_json_as_string>
can be used instead of
cfy secrets create gcp_credentials  -f <path_to_gcp_service_acoount_json>
.

Terminology

Types

The following are node type definitions. Nodes describe resources in your cloud infrastructure. For more information, see node type.

Common Properties

All cloud resource nodes have common properties:

Properties

Every time you manage a resource with Studio Conductor, it creates one or more connections to the GCP API. You specify the configuration for these clients using the gcp_config property.

It should be a dictionary, with the following values:

Example

...
node_types:
  my_vm:
    type: cloudify.gcp.nodes.Instance
    properties:
      image_id: <GCP image ID>
      gcp_config:
        project: a-gcp-project-123456
        zone: us-east1-b
        auth: <GCP auth JSON file>

Using Existing Resources

All GCP Studio Conductor node types have the properties use_external_resource and resource_id. If use_external_resource is set to true, then the plugin will attempt to locate the resource specified in resource_id and use it. An older method is still supported: the required properties for that type name, and sometimes region or zone, are used to look up an existing resource in the GCP project. If the entity is discovered, its data is used to populate the Studio Conductor instance’s attributes (runtime_properties). If it is not found, the blueprint fails to deploy.

This behavior is common to all resource types that support use_external_resource:

Runtime Properties

See section on runtime properties

Most node types will write a snapshot of the resource information from GCP when the node creation has finished (some, e.g. DNSRecord don’t correspond directly to an entity in GCP, so this is not universal).

Node Types

cloudify.gcp.nodes.Address

Derived From: cloudify.gcp.nodes.GlobalAddress

A GCP Address. This can be connected to a cloudify.gcp.nodes.Instance type using the cloudify.gcp.relationships.instance_connected_to_ip relationship type.

Properties:

cloudify.gcp.nodes.BackendService

Derived From: cloudify.nodes.Root

A group of cloudify.gcp.nodes.Instance types (contained within InstanceGroups) that can be used as the back end for load balancing.

Properties:

cloudify.gcp.nodes.DNSAAAARecord

Derived From: cloudify.gcp.nodes.DNSRecord

AAAA type DNSRecord

Properties:

cloudify.gcp.nodes.DNSMXRecord

Derived From: cloudify.gcp.nodes.DNSRecord

MX type DNSRecord

Properties:

cloudify.gcp.nodes.DNSNSRecord

Derived From: cloudify.gcp.nodes.DNSRecord

NS type DNSRecord

Properties:

cloudify.gcp.nodes.DNSRecord

Derived From: cloudify.nodes.Root

Corresponds to a specific subdomain (or @ for the root) and record-type in the containing DNSZone.

for example, the A record for special_service.getcloudify.org.

A number of convenience types are provided that update the default type (see DNSAAAARecord, DNSMXRecord, DNSTXTRecord, DNSNSRecord).

Properties:

Example

www:
  type: cloudify.gcp.nodes.DNSRecord
  properties:
    resources: [10.11.12.13, 8.9.10.11]
  relationships:
    - type: cloudify.gcp.relationships.dns_record_contained_in_zone
      target: my_zone

mx:
  type: cloudify.gcp.nodes.DNSMXRecord
  properties:
    name: mail
  relationships:
    - type: cloudify.gcp.relationships.dns_record_contained_in_zone
      target: my_zone
    - type: cloudify.gcp.relationships.dns_record_connected_to_instance
      target: my_instance

The DNSRecord type can be connected to a cloudify.gcp.nodes.Instance (using cloudify.gcp.relationships.dns_record_connected_to_instance), or to a cloudify.gcp.nodes.IPAddress (using cloudify.gcp.relationships.dns_record_connected_to_ip). Each instance or IP address node that is connected will have its public IP address added to the list of resources for this record.

cloudify.gcp.nodes.DNSTXTRecord

Derived From: cloudify.gcp.nodes.DNSRecord

TXT type DNSRecord

Properties:

cloudify.gcp.nodes.DNSZone

Derived From: cloudify.nodes.Root

A Cloud DNS zone.

Represents a specific DNS domain that you want to manage through Google Cloud DNS. DNS nameservers can vary between different DNSZones. To find the correct nameserver entries for your domain, use the nameServers attribute from the created zone.

Properties:

Example

my_zone:
  type: cloudify.gcp.nodes.DNSZone
  properties:
    dns_name: getcloudify.org.

The supplied dns_name must be a fully-qualified domain name with the trailing dot. The output attributes (runtime_properties) will include a key nameServers that contains the list of nameservers to be supplied as nameservers with the domain registrar.

cloudify.gcp.nodes.ExternalIP

Derived From: cloudify.nodes.VirtualIP

Use this together with the cloudify.gcp.relationships.instance_connected_to_ip if you want the instance to have an ephemeral external IP address.

Properties:

cloudify.gcp.nodes.FirewallRule

Derived From: cloudify.nodes.Root

A GCP FirewallRule.

This describes permitted traffic that is directed to either the entire specified network, or to Instances specified by matching tags.

Properties:

Example

allow_ssh:
  type: cloudify.gcp.nodes.FirewallRule
  properties:
    sources: [0.0.0.0/0]
    allowed:
      tcp: [22]

allow_http_to_http_tag:
  type: cloudify.gcp.nodes.FirewallRule
  properties:
    sources: [0.0.0.0/0]
    allowed:
      tcp: [80]
    target_tags: [http]

http_instance:
  type: cloudify.gcp.nodes.Instance
  properties:
    tags: [http]
    ...

cloudify.gcp.nodes.GlobalAddress

Derived From: cloudify.nodes.VirtualIP

A GCP GlobalAddress.

You can only use GlobalAddress together with GlobalForwardingRule. To connect a static IP address to an Instance, use StaticIP.

Properties:

cloudify.gcp.nodes.GlobalForwardingRule

Derived From: cloudify.nodes.Root

A GCP GlobalForwardingRule.

Can only be used in conjunction with a GlobalAddress to set up HTTP and HTTPS forwarding.

Properties:

cloudify.gcp.nodes.HealthCheck

Derived From: cloudify.nodes.Root

A GCP HealthCheck.

This describes a method that a TargetProxy can use to verify that specific backend Instances are functioning. Backends that fail the health check verification are removed from the list of candidates.

Properties:

cloudify.gcp.nodes.Image

Derived From: cloudify.nodes.Root

A stored image that can be used as the base for new Instances.

Properties:

cloudify.gcp.nodes.Instance

Derived From: cloudify.nodes.Compute

A GCP Instance (i.e. a VM).

Properties:

Example

my_gcp_instance:
  type: cloudify.gcp.nodes.Instance
  properties:
    image_id: http://url.to.your.example.com/image
    instance_type: n1-standard-1
    gcp_config:
      project: your-project
      network: default
      zone: us-east1-b
      auth: path_to_auth_file.json

This example includes actions of adding additional parameters, tagging an instance name, and explicitly defining the gcp_config.

cloudify.gcp.nodes.InstanceGroup

Derived From: cloudify.nodes.Root

A GCP InstanceGroup. This is used to configure failover systems. You can configure InstanceGroups to scale automatically, based on load, and to replace failing Instances with freashly started ones.

Properties:

cloudify.gcp.nodes.KeyPair

Derived From: cloudify.nodes.Root

An SSH key-pair which will be uploaded to any Instances connected to it via cloudify.gcp.relationships.instance_connected_to_keypair.

Unlike other cloud providers, users are dynamically created on Instances based on the username specified by the uploaded SSH key, so the public key text must include a username in the comment section (keys generated using ssh-keygen have this by default).

Properties:

cloudify.gcp.nodes.Network

Derived From: cloudify.nodes.Network

A GCP Network. This supports either auto-assigned or manual subnets. Legacy networks are not supported. See the GCP Manager and Networks section below if you plan to run a Conductor Manager on GCP.

Properties:

Example

my_net:
  type: cloudify.gcp.nodes.Network

cloudify.gcp.nodes.Route

Derived From: cloudify.nodes.Router

A defined route, which will be added to the specified network. If tags are specified, it will only be added to Instances matching them.

Properties:

cloudify.gcp.nodes.SecurityGroup

Derived From: cloudify.nodes.SecurityGroup

A virtual SecurityGroup.

Google Cloud Platform has no entity equivalent to a Security Group on AWS or OpenStack, so as a convenience Studio Conductor includes a virtual one. It is implemented behind the scenes using a specially constructed tag and a number of FirewallRules.

Properties:

cloudify.gcp.nodes.SslCertificate

Derived From: cloudify.nodes.Root

A TLS/SSL certificate and key. This will be used by a HTTPS TargetProxy to provide authenticated encryption for connecting users.

Properties:

cloudify.gcp.nodes.StaticIP

Derived From: cloudify.gcp.nodes.GlobalAddress

Alias for GlobalAddress for backward compatibility.

Properties:

cloudify.gcp.nodes.SubNetwork

Derived From: cloudify.nodes.Subnet

A GCP Subnetwork. Must be connected to a Network using cloudify.gcp.relationships.contained_in_network.

Only networks with the auto_subnets property disabled can be used.

Properties:

Example

my_net:
  type: cloudify.gcp.nodes.Network
  properties:
    auto_subnets: false

my_subnet:
  type: cloudify.gcp.nodes.SubNetwork
  properties:
    subnet: 10.8.0.0/20
  relationships:
    - type: cloudify.gcp.relationships.contained_in_network
      target: my_net

my_instance:
  type: cloudify.gcp.nodes.Instance
  properties:
    ...
  relationships:
    - type: cloudify.gcp.relationships.contained_in_network
      target: my_subnet

If you want to use an exsisting SubNetwork (use_external_resource: true) then you must supply the name and region properties. This is because SubNetwork names are not unique across the whole project, only within a region.

cloudify.gcp.nodes.TargetProxy

Derived From: cloudify.nodes.Root

A TargetHttpProxy or TargetHttpsProxy.

Specify which using the target_proxy_type property.

Properties:

cloudify.gcp.nodes.UrlMap

Derived From: cloudify.nodes.Root

Maps URLs to BackendServices

Properties:

cloudify.gcp.nodes.Volume

Derived From: cloudify.nodes.Volume

A GCP Volume.

A virtual disk which can be attached to Instances.

Properties:

cloudify.gcp.nodes.KubernetesCluster

Derived From: cloudify.nodes.Root

This node type refers to a GCP GKE Cluster.

Properties:

Cluster Example

Creates a new GKE Cluster

  kubernetes-cluster:
    type: cloudify.gcp.nodes.KubernetesCluster
    properties:
      name: { concat: [ { get_input: resource_prefix }, '-cluster']}
      gcp_config: *gcp_config

cloudify.gcp.nodes.KubernetesNodePool

Derived From: cloudify.nodes.Root

This node type reefers to Node pool in a GKE cluster.

Properties:

Nodepool Example

  kubernetes-cluster-node-pool:
    type: cloudify.gcp.nodes.KubernetesNodePool
    properties:
      name: { concat: [ { get_input: resource_prefix }, '-node-pool-1']}
      cluster_id: { get_property: [ kubernetes-cluster, name] }
      additional_settings:
        config:
          machineType: n1-standard-2
        initialNodeCount: 2
        autoscaling:
          enabled: true
          minNodeCount: 2
          maxNodeCount: 5
      gcp_config: *gcp_config
    relationships:
      - type: cloudify.relationships.depends_on
        target: kubernetes-cluster

cloudify.gcp.nodes.KubernetesClusterMonitoring

Derived From: cloudify.nodes.Root

This node type refers to a GKE cluster monitoring service.

Properties:

cloudify.gcp.nodes.KubernetesClusterNetworkPolicy

This node type refers to a GKE cluster network policy.

Derived From: cloudify.nodes.Root

Properties:

Relationships

cloudify.gcp.relationships.contained_in_compute

Derived From: cloudify.relationships.contained_in

cloudify.gcp.relationships.contained_in_network

Derived From: cloudify.relationships.contained_in

cloudify.gcp.relationships.dns_record_connected_to_instance

Derived From: cloudify.relationships.connected_to

cloudify.gcp.relationships.dns_record_connected_to_ip

Derived From: cloudify.relationships.connected_to

cloudify.gcp.relationships.dns_record_contained_in_zone

Derived From: cloudify.relationships.contained_in

cloudify.gcp.relationships.file_system_contained_in_compute

Derived From: cloudify.relationships.contained_in

cloudify.gcp.relationships.forwarding_rule_connected_to_target_proxy

Derived From: cloudify.relationships.connected_to

cloudify.gcp.relationships.instance_connected_to_disk

Derived From: cloudify.relationships.connected_to

cloudify.gcp.relationships.instance_connected_to_instance_group

Derived From: cloudify.relationships.connected_to

cloudify.gcp.relationships.instance_connected_to_ip

Derived From: cloudify.relationships.connected_to

cloudify.gcp.relationships.instance_connected_to_keypair

Derived From: cloudify.relationships.connected_to

cloudify.gcp.relationships.instance_connected_to_security_group

Derived From: cloudify.relationships.connected_to

cloudify.gcp.relationships.instance_contained_in_network

Derived From: cloudify.relationships.contained_in

cloudify.gcp.relationships.uses_as_backend

Derived From: cloudify.relationships.connected_to

Account Information

The plugin needs access to your GCP auth credentials (via the gcp_config parameter) in order to operate (but see below about use within a manager).

gcp_config

If you don’t want to provide the gcp_config dictionary to every node in your blueprints, you can provide it, as json, at /etc/cloudify/gcp_plugin/gcp_config

Networks

Instances in GCP are not able to communicate internally with instances in a different network. This means that if you want to run Studio Conductor agents on your nodes they must be in the same network as the manager.

Additionally, a given network must choose either auto-subnets or manual subnets operation when created. For maximum flexibility, auto_subnets: false is recommended, though this requires that subnets are created for any region you wish to place Instances in.