PCF on Azure with Platform Automation using PCF-Pipelines - PART 1

Thats a loaded set of words. Many of our customers at Pivotal have started looking at Data Center augmentation, specially for burst needs using a public IAAS provider like AWS , GCP and Azure. In this edition of my blog , I wanted to go over the process of installing a full Pivotal Cloud Foundry Foundation.  Even though we at Pivotal provide detail instructions for a manual install of the foundation,  automation has always been a big need.

Concourse and Platform Automation
Concourse is the CI tool developed at Pivotal for all our automation needs during the Cloud foundry development. With the mindset of treating "Platform as a Product" and also eating our own dogfood ( Wait ,  should I say "drinking our own champagne"), we have a new product slug called PCF-Pipelines. It is a opinionated set of pipelines that help automate the whole process of installing, upgrading and maintaining all the tiles on the platform.

About the Trail 
Having gone through the install journey as I prep to work with multiple customers,  I thought I will throw the breadcrumbs to help document the Trail! In Part 1 of this multipart  blog, I will focus on installing bosh via bosh boot loader. I will come back and talk about installing and configuring concourse with a couple simple examples to see it working and  PCF installation using the pipelines on subsequent blogs.

Lets install a copy of bosh that will control concourse and a jumpbox to access the components installed.

The BBL docs explains how to do this for aws, azure and gcp.  I have picked up the info needed for azure below.
  1. Install bbl ( the boot loader ) 

  2. $ brew tap cloudfoundry/tap
    $ brew install bosh-cli
    $ brew install bbl


  3.  Setup the Environment variables via a .PS1 file as below to allow easy reuse. 
  4. (change the value that says "CHANGEME" to real values based on your IAAS info)

    anandrao at Anands-MBP in ~/pivotal/repos/bbl-az
    $ cat bbl-az-variables.PS1
    export BBL_IAAS=azure
    export BBL_AZURE_CLIENT_ID="CHANGEME"
    export BBL_AZURE_CLIENT_SECRET="CHANGEME"
    export BBL_AZURE_REGION=westus
    export BBL_AZURE_SUBSCRIPTION_ID="CHANGEME"
    export BBL_AZURE_TENANT_ID=
    "CHANGEME"

  5. Execute the below commands to setup your Bosh along with a jumpbox and yea a load balancer as well. "bbl" does it all in one command.  ( I would create a folder and run bbl inside that as  all the files bbl creates will be clearly in one distinct place).


    anandrao at Anands-MBP in ~/pivotal/repos/bbl-az
    $ mkdir bbl ; cd bbl

    anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/bbl
    $ bbl plan
    step: generating terraform template
    step: generating terraform variables
    step: terraform init



    anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/bbl
    $ bbl up
    step: terraform init
    step: terraform apply
    step: creating jumpbox
    Deployment manifest: '/Users/anandrao/pivotal/repos/bbl-az/bbl/jumpbox-deployment/jumpbox.yml'
    Deployment state: '/Users/anandrao/pivotal/repos/bbl-az/bbl/vars/jumpbox-state.json'

    Started validating
    Downloading release 'os-conf'... Skipped [Found in local cache] (00:00:00)
    Validating release 'os-conf'... Finished (00:00:00)
    Downloading release 'bosh-azure-cpi'... Skipped [Found in local cache] (00:00:00)
    Validating release 'bosh-azure-cpi'... Finished (00:00:00)
    Validating cpi release... Finished (00:00:00)
    Validating deployment manifest... Finished (00:00:00)
    Downloading stemcell... Skipped [Found in local cache] (00:00:00)
    Validating stemcell... Finished (00:00:01)
    Finished validating (00:00:01)

    Started installing CPI
    Compiling package 'ruby-2.4-r3/8471dec5da9ecc321686b8990a5ad2cc84529254'... Finished (00:02:33)
    Compiling package 'bosh_azure_cpi/ceb90b730e4e350787d1be2b81bb97b433549f3f'... Finished (00:01:01)
    Installing packages... Finished (00:00:00)
    Rendering job templates... Finished (00:00:00)
    Installing job 'azure_cpi'... Finished (00:00:00)
    Finished installing CPI (00:03:36)

    Starting registry... Finished (00:00:00)
    Uploading stemcell 'bosh-azure-hyperv-ubuntu-trusty-go_agent/3468.17'... Finished (00:01:42)

    Started deploying
    Creating VM for instance 'jumpbox/0' from stemcell 'bosh-stemcell-b6c82a4f-e14f-4ef9-91f6-f21339fc0b15'... Finished (00:01:36)
    Waiting for the agent on VM 'agent_id:22df9d2e-e05d-4ae6-5f5f-622d66debf6d;resource_group_name:bbl-env-ontario-2018-07-31t05-15z-bosh;storage_account_name:bblenvontario2018073jfde' to be ready... Finished (00:00:25)
    Rendering job templates... Finished (00:00:00)
    Updating instance 'jumpbox/0'... Finished (00:00:12)
    Waiting for instance 'jumpbox/0' to be running... Finished (00:00:00)
    Running the post-start scripts 'jumpbox/0'... Finished (00:00:00)
    Finished deploying (00:02:21)

    Stopping registry... Finished (00:00:00)
    Cleaning up rendered CPI jobs... Finished (00:00:00)

    Succeeded
    step: created jumpbox
    step: creating bosh director
    Deployment manifest: '/Users/anandrao/pivotal/repos/bbl-az/bbl/bosh-deployment/bosh.yml'
    Deployment state: '/Users/anandrao/pivotal/repos/bbl-az/bbl/vars/bosh-state.json'

    Started validating
    Downloading release 'bosh'... Finished (00:00:08)
    Validating release 'bosh'... Finished (00:00:00)
    Downloading release 'bpm'... Finished (00:00:11)
    Validating release 'bpm'... Finished (00:00:00)
    Downloading release 'bosh-azure-cpi'... Finished (00:00:05)
    Validating release 'bosh-azure-cpi'... Finished (00:00:00)
    Downloading release 'os-conf'... Skipped [Found in local cache] (00:00:00)
    Validating release 'os-conf'... Finished (00:00:00)
    Downloading release 'uaa'... Finished (00:01:16)
    Validating release 'uaa'... Finished (00:00:00)
    Downloading release 'credhub'... Finished (00:00:15)
    Validating release 'credhub'... Finished (00:00:00)
    Validating cpi release... Finished (00:00:00)
    Validating deployment manifest... Finished (00:00:00)
    Downloading stemcell... Finished (00:00:47)
    Validating stemcell... Finished (00:00:01)
    Finished validating (00:02:52)

    Started installing CPI
    Compiling package 'ruby-2.4-r3/8471dec5da9ecc321686b8990a5ad2cc84529254'... Finished (00:02:30)
    Compiling package 'bosh_azure_cpi/e83f4474b88d5f34304ce99a0e1cead2e2ae3627'... Finished (00:01:03)
    Installing packages... Finished (00:00:00)
    Rendering job templates... Finished (00:00:00)
    Installing job 'azure_cpi'... Finished (00:00:00)
    Finished installing CPI (00:03:35)

    Starting registry... Finished (00:00:00)
    Uploading stemcell 'bosh-azure-hyperv-ubuntu-trusty-go_agent/3586.24'... Finished (00:01:36)

    Started deploying
    Creating VM for instance 'bosh/0' from stemcell 'bosh-stemcell-c37f79e2-8cbc-459f-b2c7-8f099707ef8d'... Finished (00:02:38)
    Waiting for the agent on VM 'agent_id:7d2deaab-d7db-4adf-711d-f05f3adbfff2;resource_group_name:bbl-env-ontario-2018-07-31t05-15z-bosh;storage_account_name:bblenvontario2018073jfde' to be ready... Finished (00:00:24)
    Creating disk... Finished (00:00:36)
    Attaching disk 'caching:None;disk_name:bosh-data-9d98f6ea-399c-41b1-888c-86439cf86100;storage_account_name:bblenvontario2018073jfde' to VM 'agent_id:7d2deaab-d7db-4adf-711d-f05f3adbfff2;resource_group_name:bbl-env-ontario-2018-07-31t05-15z-bosh;storage_account_name:bblenvontario2018073jfde'... Finished (00:01:06)
    Rendering job templates... Finished (00:00:11)
    Compiling package 'ruby-2.4-r4/0cdc60ed7fdb326e605479e9275346200af30a25'... Skipped [Package already compiled] (00:00:00)
    Compiling package 'mysql/898f50dde093c366a644964ccb308a5281c226de'... Skipped [Package already compiled] (00:00:00)
    Compiling package 'libpq/e2414662250d0498c194c688679661e09ffaa66e'... Skipped [Package already compiled] (00:00:00)
    Compiling package 'bpm-runc/c0b41921c5063378870a7c8867c6dc1aa84e7d85'... Skipped [Package already compiled] (00:00:00)
    Compiling package 'golang/e3ca1c9440c29ad576d633e9ef6a2f7805a5e8b7'... Skipped [Package already compiled] (00:00:07)
    Compiling package 'ruby-2.4-r3/8471dec5da9ecc321686b8990a5ad2cc84529254'... Finished (00:02:12)
    Compiling package 'openjdk_1.8.0/c8846344bf802835ce8b1229de8fa2028d06f603'... Skipped [Package already compiled] (00:00:02)
    Compiling package 'golang-1.9-linux/8d6c67abda8684ce454f0bc74050a213456573ff'... Skipped [Package already compiled] (00:00:06)
    Compiling package 'gonats/73ec55f11c24dd7c02288cdffa24446023678cc2'... Skipped [Package already compiled] (00:00:00)
    Compiling package 's3cli/3097f27cb9356172c9ae52de945821c4e338c87a'... Skipped [Package already compiled] (00:00:00)
    Compiling package 'uaa/cdb6217bd1b700002b9746c0b069d79480edb192'... Skipped [Package already compiled] (00:00:09)
    Compiling package 'verify_multidigest/8fc5d654cebad7725c34bb08b3f60b912db7094a'... Skipped [Package already compiled] (00:00:00)
    Compiling package 'registry/a6daac4743749c70c2ae15e58170adb6b41a3a76'... Skipped [Package already compiled] (00:00:01)
    Compiling package 'health_monitor/251915bca2d42f06f4bbb1f5395afd1ae73cf681'... Skipped [Package already compiled] (00:00:00)
    Compiling package 'director/db07ae48ea2963a9cdec4938a9522f85f718e672'... Skipped [Package already compiled] (00:00:01)
    Compiling package 'postgres-9.4/52b3a31d7b0282d342aa7a0d62d8b419358c6b6b'... Skipped [Package already compiled] (00:00:00)
    Compiling package 'bosh-gcscli/fce60f2d82653ea7e08c768f077c9c4a738d0c39'... Skipped [Package already compiled] (00:00:00)
    Compiling package 'bpm/3fe49cfa0140be3ebd8da4bdcadfa6b84d847e87'... Skipped [Package already compiled] (00:00:00)
    Compiling package 'lunaclient/b922e045db5246ec742f0c4d1496844942d6167a'... Skipped [Package already compiled] (00:00:00)
    Compiling package 'bosh_azure_cpi/e83f4474b88d5f34304ce99a0e1cead2e2ae3627'... Finished (00:00:46)
    Compiling package 'uaa_utils/90097ea98715a560867052a2ff0916ec3460aabb'... Skipped [Package already compiled] (00:00:00)
    Compiling package 'nginx/d9f726bf0c5a38bad988e40cefb084c821e333cf'... Skipped [Package already compiled] (00:00:00)
    Compiling package 'credhub/e3d60a289d5fd414e29ee06e7e5f1a6b3802c792'... Skipped [Package already compiled] (00:00:04)
    Compiling package 'configurator/0d632a3a9b06f3777bea07d61807ca06ece24dee'... Skipped [Package already compiled] (00:00:00)
    Compiling package 'davcli/f8a86e0b88dd22cb03dec04e42bdca86b07f79c3'... Skipped [Package already compiled] (00:00:00)
    Updating instance 'bosh/0'... Finished (00:01:48)
    Waiting for instance 'bosh/0' to be running... Finished (00:02:09)
    Running the post-start scripts 'bosh/0'... Finished (00:00:20)
    Finished deploying (00:13:02)

    Stopping registry... Finished (00:00:00)
    Cleaning up rendered CPI jobs... Finished (00:00:00)

    Succeeded
    step: created bosh director
    step: generating cloud config
    step: applying cloud config


    Very cool , you just installed bosh ,  a jump host built with all the security and controls.                
  6. Lets inspect what we just installed and verify it. "bbl" has a few options that help in verification..

    Environmental Detail Commands: Useful for automation and gaining access
    jumpbox-address Prints BOSH jumpbox address
    director-address Prints BOSH director address
    director-username Prints BOSH director username
    director-password Prints BOSH director password
    director-ca-cert Prints BOSH director CA certificate
    env-id Prints environment ID
    ssh-key Prints jumpbox SSH private key
    director-ssh-key Prints director SSH private key
    lbs Prints load balancer(s) and DNS records
    outputs Prints the outputs from terraform
    ssh Opens an SSH connection to the director or jumpbox

    anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/bbl
    $ bbl director-address
    https://10.0.0.6:25555

    anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/bbl
    $ bbl director-username
    admin.


    Azure Portal View
    Azure Portal View


  7. Lets run some bosh commands and check the environment. The first step I do after the install is extract all the exports commands provided by the "print-env" option of bbl. This will allow you to use bosh and other components installed ( bosh , credhub and the jumpbox). I would inspect the output of the "bbl print-env" .

    anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/bbl
    $ bbl print-env > bbl-all-exports.PS1

    anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/bbl
    $ source bbl-all-exports.PS1

    anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/bbl
    $ bosh vms
    Using environment 'https://10.0.0.6:25555' as client 'admin'

    Succeeded

    anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/bbl
    $ bosh environment
    Using environment 'https://10.0.0.6:25555' as client 'admin'

    Name bosh-bbl-env-ontario-2018-07-31t05-15z
    UUID a03916c8-01f1-4cc2-abe5-93a546e86b79
    Version 266.4.0 (00000000)
    CPI azure_cpi
    Features compiled_package_cache: disabled
    config_server: enabled
    dns: disabled
    snapshots: disabled
    User admin

    Succeeded

  8. I would inspect some of the folders created under the bbl folder. You will see the bosh manifests created for the vms created as part of the bbl based install.

Now that we have bosh up and running,  we could try out all of our typical bosh commands. Please refer to the bosh cli docs for this. 

I will jump into installing concourse as our next step in this journey.

Quick links to the "The PCF on Azure" Blog Series :