This Blog Entry is a part 2 of a series. The Part - 1 covered the bbl and bosh installation.

When you go to the concourse-ci.org website and start looking at multiple ways to install concourse,  you start wondering whats the best practice to install concourse. Speaking to a many of the amazing folks at Pivotal I decided to follow the path of installing Bosh and having Bosh start and manage concourse. When it comes to installing Bosh, once again there are multiple ways to do this. The Bosh-Boot-loader project comes in very handy to get your bosh instance up and running in the quickest time.

With the successful install of bbl and bosh as part of the Part-1 of this series , lets dive into the installation of concourse using the bosh instance that we have running.

The first step in this process is downloading ( cloning from github) a copy of the concourse-bosh-deployment.

anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/bbl
$ cd ..

anandrao at Anands-MBP in ~/pivotal/repos/bbl-az
$ mkdir concourse ; cd concourse

anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/concourse
$ git clone https://github.com/concourse/concourse-bosh-deployment.git
Cloning into 'concourse-bosh-deployment'...
remote: Counting objects: 705, done.
remote: Compressing objects: 100% (79/79), done.
remote: Total 705 (delta 58), reused 58 (delta 26), pack-reused 600
Receiving objects: 100% (705/705), 121.01 KiB | 1.66 MiB/s, done.
Resolving deltas: 100% (384/384), done.
Next , we will need to find out the version of OS/Stemcell that is referenced in the concourse.yml file. On the Latest version as of today, here is what I found:

stemcells:
- alias: xenial
os: ubuntu-xenial
version: latest

Ok , ubuntu-xenial it is. Remember , it used to trusty before. Armed with this info, lets hunt down the stemcell for this. Best place to find this will be https://bosh.io/stemcells.


Using the command on that page ,  lets upload the stemcell to bosh.  

anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/concourse/concourse-bosh-deployment/cluster (master)
$ bosh upload-stemcell --sha1 1d660dcbb51a1c80914fbb8c11173e640014f6c5 \
https://bosh.io/d/stemcells/bosh-azure-hyperv-ubuntu-xenial-go_agent\?v\=97.3
Using environment 'https://10.0.0.6:25555' as client 'admin'

Task 9

Task 9 | 07:09:48 | Update stemcell: Downloading remote stemcell (00:00:15)
Task 9 | 07:10:03 | Update stemcell: Verifying remote stemcell (00:00:02)
Task 9 | 07:10:05 | Update stemcell: Extracting stemcell archive (00:00:04)
Task 9 | 07:10:10 | Update stemcell: Verifying stemcell manifest (00:00:00)
Task 9 | 07:11:20 | Update stemcell: Checking if this stemcell already exists (00:00:00)
Task 9 | 07:11:20 | Update stemcell: Uploading stemcell bosh-azure-hyperv-ubuntu-xenial-go_agent/97.3 to the cloud (00:02:54)
Task 9 | 07:14:14 | Update stemcell: Save stemcell bosh-azure-hyperv-ubuntu-xenial-go_agent/97.3 (bosh-stemcell-05475b0d-eca0-4666-839b-07b2db8a80c4) (00:00:00)

Task 9 Started Tue Jul 31 07:09:48 UTC 2018
Task 9 Finished Tue Jul 31 07:14:14 UTC 2018
Task 9 Duration 00:04:26
Task 9 done

Succeeded

Ok,  the Stemcell is in, lets run the bosh magic to get concourse running. One step we need to do as a bbl step here is to add a load-balancer for concourse. Remember,  we did not add this in our initial infra creation.  Its easy to update out bbl plan and re-apply. 

Switch back to the bbl folder and exec "bbl plan --lb-type concourse" as below and then run "bbl up".


anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/bbl
$ pwd
/Users/anandrao/pivotal/repos/bbl-az/bbl

anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/bbl
$ ls
bbl-all-exports.PS1 bosh-deployment create-director.sh delete-director.sh jumpbox-deployment vars
bbl-state.json cloud-config create-jumpbox.sh delete-jumpbox.sh terraform

anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/bbl
$ bbl plan --lb-type concourse
step: generating terraform template
step: generating terraform variables
step: terraform init

anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/bbl
$ bbl up
step: terraform init
step: terraform apply
step: creating jumpbox
Deployment manifest: '/Users/anandrao/pivotal/repos/bbl-az/bbl/jumpbox-deployment/jumpbox.yml'
Deployment state: '/Users/anandrao/pivotal/repos/bbl-az/bbl/vars/jumpbox-state.json'

Started validating
Downloading release 'os-conf'... Skipped [Found in local cache] (00:00:00)
Validating release 'os-conf'... Finished (00:00:00)
Downloading release 'bosh-azure-cpi'... Skipped [Found in local cache] (00:00:00)
Validating release 'bosh-azure-cpi'... Finished (00:00:00)
Validating cpi release... Finished (00:00:00)
Validating deployment manifest... Finished (00:00:00)
Downloading stemcell... Skipped [Found in local cache] (00:00:00)
Validating stemcell... Finished (00:00:01)
Finished validating (00:00:01)
No deployment, stemcell or release changes. Skipping deploy.

Succeeded
step: created jumpbox
step: creating bosh director
Deployment manifest: '/Users/anandrao/pivotal/repos/bbl-az/bbl/bosh-deployment/bosh.yml'
Deployment state: '/Users/anandrao/pivotal/repos/bbl-az/bbl/vars/bosh-state.json'

Started validating
Downloading release 'bosh'... Skipped [Found in local cache] (00:00:00)
Validating release 'bosh'... Finished (00:00:00)
Downloading release 'bpm'... Skipped [Found in local cache] (00:00:00)
Validating release 'bpm'... Finished (00:00:00)
Downloading release 'bosh-azure-cpi'... Skipped [Found in local cache] (00:00:00)
Validating release 'bosh-azure-cpi'... Finished (00:00:00)
Downloading release 'os-conf'... Skipped [Found in local cache] (00:00:00)
Validating release 'os-conf'... Finished (00:00:00)
Downloading release 'uaa'... Skipped [Found in local cache] (00:00:00)
Validating release 'uaa'... Finished (00:00:00)
Downloading release 'credhub'... Skipped [Found in local cache] (00:00:00)
Validating release 'credhub'... Finished (00:00:00)
Validating cpi release... Finished (00:00:00)
Validating deployment manifest... Finished (00:00:00)
Downloading stemcell... Skipped [Found in local cache] (00:00:00)
Validating stemcell... Finished (00:00:01)
Finished validating (00:00:04)
No deployment, stemcell or release changes. Skipping deploy.

Succeeded
step: created bosh director
step: generating cloud config
step: applying cloud config

The Load-Balancer is ready to go. You could check this by running "bbl lbs"
Next,  we will switch back to the concourse folder  and execute the command to deploy concourse. Make sure to be in the right folder when you run this command as there are file references  in the command.


anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/concourse
$ cd concourse-bosh-deployment/cluster

anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/concourse/concourse-bosh-deployment/cluster (master)
$ pwd
/Users/anandrao/pivotal/repos/bbl-az/concourse/concourse-bosh-deployment/cluster

Lets look at the bosh command that is used to deploy concourse as a bosh-deployment.

bosh deploy -d concourse concourse.yml \
-l ../versions.yml \
--vars-store cluster-creds.yml \
-o operations/basic-auth.yml \
-o operations/privileged-http.yml \
-o operations/privileged-https.yml \
-o operations/tls.yml \
-o operations/tls-vars.yml \
-o operations/web-network-extension.yml \
--var network_name=default \
--var external_url='https://wings.az.clue2solve.com' \
--var external_host='wings.az.clue2solve.com' \
--var web_vm_type=default \
--var db_vm_type=default \
--var db_persistent_disk_type=10GB \
--var worker_vm_type=default \
--var deployment_name=concourse \
--var web_network_name=private \
--var local_user.username=admin \
--var local_user.password='$2a$10$CjHFkbuSssxtqtBfqLZ.MuwA0BP589fUd4rXNIM4HwIQ4Dw00NoRq' \
--var web_network_vm_extension=lb


Few things we need to be careful about here:

  • The external_url should be a valid domain and make sure to include the https://  for this value
  • The external_host should be a common name ( dns that is valid )  without the https
  • the local_user.password should be a becrypted value of the password. There is a strong requirement that the password should "bcrypted" at least 10 rounds. I used the website https://www.browserling.com/tools/bcrypt and used the generated hash ( make sure you select 10 rounds there  ) as the password. 

Gist here shows the command execution and the output.

Hurray ,  we successfully deployed concourse. Lets get the DNS Entry for wings ( concourse ) updated with the load balancer IP. you could do this by

anandrao at Anands-MBP in ~/pivotal/repos/bbl-az/bbl
$ bbl lbs
Concourse LB: bbl-env-ontario-2018-07-31t05-15z-concourse-lb (20.189.131.186)

Update (create one if not already there) the A-Record on your DNS Zone with the IP.

And Enjoy Concourse !


You can create a test pipeline and execute it to test. I suggest starting with https://concoursetutorial.com. It is an awesome tutorial to get into concourse.

A sample Screen with successful concourse pipelines are below.




In the Next Part ( Part 3 )  of this Series , I will cover the PCF installation using the concourse instance that we just installed.


Quick links to the "The PCF on Azure" Blog Series :