Google Cloud Build
A DevOps Tale
Contents
Preamble
Imagine a world where after you write a piece of code, you need to teach your operations team how the code works, how to implement it, how to test it and how to support it.
In 2007-2008 (earlier than you probably thought) it emerged that this was a fatal flaw of the software development industry, each of these teams - development and operations - grew independently with competing goals and metrics for success.
Something happened around that time in forums and a few leaders in the industry across both sides of the Development and Operations aisle. People wanted to flip this on its head and what we have now is the thing that we call DevOps.
DevOps in the view of Atlassian’s software suite
DevOps Benefits
You’ve probably heard of the magic of DevOps and why nearly all (99%) of DevOps teams are confident about the success of their code that goes into production (based on a survey of 500 DevOps practitioners conducted by Atlassian).
The 2019 State of DevOps Report is based on research that provides the most comprehensive view of the growing DevOps industry.
According to this report, elite DevOps performers:
-
Deploy code 208 times more frequently than low performers
-
Have 106 times faster change lead time from commit to deploy than low performers
-
Change failure rates that are seven times better than low performers
-
Restore service 2,604 times faster than low performers
(Numbers based on the DORA DevOps Report 2019)
These are all obviously enormous numbers… but what does it mean? And how do I do DevOps? The good news is there’s no magic trick to DevOps - and like most good things its the result of strategic and incremental changes.
The CALMS Framework
The CALMS framework is a way of measuring success and evaluation adoption of DevOps practices in an organisation. The acronym was coined by Jez Humble, co-author of “The DevOps Handbook,” and stands for Culture, Automation, Lean, Measurement, and Sharing.
-
Culture
- Product based teams that work together to fuel a release rather than function based teams that just focus on getting “their job done”
-
Automation
- Automation helps eliminate repetitive manual work, yields repeatable processes, and creates reliable systems.
-
Lean
-
Based on lean manufacturing philosophies born out of Toyota in Japan.
-
Lean philosophy regards everything not adding value to the customer as waste. Such waste may include
-
Partially done work
-
Extra features
-
Relearning
-
Task switching
-
Waiting
-
Handoffs
-
Defects
-
Management activities
-
-
-
Measurement
-
It’s hard to prove your continuous improvement efforts actually improve anything without data. Fortunately, there are loads of metrics and tools that you can use to generate information about your progress. Some examples of useful ones:
-
How often do recurring bugs or failures happen?
-
How long does it take to recover after a system failure?
-
How long did it take to go from development to deployment?
-
-
-
Sharing
- The long-standing friction between development and operations teams is largely due to a lack of common ground. Sharing responsibility (and success) goes a long way toward bridging that divide. DevOps is big on the idea that the same people who build an application should be involved in shipping and running it. For DevOps to succeed barriers must be eliminated by adopting the “you build it, you run it” practice.
The meat of each of these pillars is important and feed wholly into the Riley flywheel, but what we want to focus on today is the second of these framework items - automation.
Automation Best Practices
Automation is essential to DevOps success. Instead of requiring humans to manually check code into a repository, run tests throughout the development process, and deploy code to production, build Docker images, deploy your CloudFormation or Terraform code, DevOps practitioners streamline this process by automating each step.
Shift left with CI/CD
When teams “shift left”, they bring testing, security and any other component into their code development processes early. Instead of sending multiple changes to a separate test or QA team, a variety of tests are performed throughout the coding process so that developers can fix bugs or improve code quality while they work on the relevant section of the codebase. The practice of continuous integration and continuous delivery (CI/CD), and deployment underpins the ability to shift left.
What’s the difference between the two CDs? And what’s a CI?
-
Continuous integration
- Developers & engineers practicing continuous integration merge their changes back to the main branch as often as possible. The developer’s changes are validated by creating a build and running automated tests against the build. By doing so, you avoid integration challenges that can happen when waiting for release day to merge changes into the release branch.
-
Continuous delivery
-
Continuous delivery is an extension of continuous integration since it automatically deploys all code changes to a testing and/or production environment after the build & test stage.
-
This means that on top of automated testing, you have an automated release process and you can deploy your application any time by clicking a button.
-
-
Continuous deployment
- Continuous deployment goes one step further than continuous delivery. With this practice, every change that passes all stages of your production pipeline is released to your customers. There’s no human intervention, and only a failed test will prevent a new change to be deployed to production.
Build with the right tools
A DevOps toolchain requires the right tools for each phase of the DevOps lifecycle, with key capabilities to improve software quality and speed of delivery, and the tool we want to talk about today is Google Cloud Build.
Build, test, and deploy on our Google’s serverless CI/CD platform.
Google Cloud Build
-
Extremely fast builds
- Runs on Google’s global network to and can choose between builds on high-CPU VMs or cache source code, images, or other dependencies to further increase your build speed.
-
Automate your deployments
- Create pipelines as a part of your build steps to automate deployments. Deploy using built-in integrations to Google Kubernetes Engine, App Engine, Cloud Functions, and Firebase.
-
Support for multicloud
- Deploy to multiple clouds as a part of your CI/CD pipeline. Cloud Build comes with builder images which have languages and tools already installed. Likewise, containerised tasks of Cloud Build are fully portable across different clouds.
-
Commit to deploy in minutes
- Set up triggers to automatically build, test, or deploy source code when you push changes to anything like a GitHub, Cloud Source Repositories, or a Bitbucket repository - or simply straight from your code folder!
-
Unparalleled privacy
- Run builds on infrastructure protected by Google Cloud security. Trigger fully managed CI/CD workflows from private source code repositories hosted in private networks, including GitHub Enterprise. With private pools you can run builds in a VPC of your choosing also
Oh King DevOps, how too can I perform a Cloud Build?
Cloud Build is a simply service that executes your builds on Google Cloud Platform’s infrastructure.
Cloud Build can import source code from a variety of repositories or cloud storage spaces, execute a build to your specifications, and produce artifacts such as Docker containers or Java archives.
You essentially write a build config (in YAML) to provide instructions to Cloud Build on what tasks to perform. You can configure builds to fetch dependencies, run unit tests, perform linting, perform static analyses, integration tests, and create artifacts with build tools such as docker, gradle, maven, bazel, terraform and CloudFormation
Example Cloud Build Configuration
Here is a simple example of a Docker image being published to Google Cloud Container Registry:
steps: - name: ‘gcr.io/cloud-builders/docker’ args: [‘build’, ‘-t’, ‘gcr.io/my-project/my-image’, ’.’] images: [‘gcr.io/my-project/my-image’]
Solving Everyday Problems
3 challenges I’ve solved with Cloud Build this past month.
-
This past week I wanted to deploy a copy of the Striim software to a customer’s account. But didn’t want to run up a VM, setup an external IP, install the software and configure. There instead exists a Docker image for Striim which we could simply publish to Container Registry, and then run on Cloud Run.
-
Ben, Tim and I were asking each other whether there was a better way to run CloudSploit across a customer’s account and due to the existence of a Docker Image for CloudSploit this was easy enough to setup, test and configure and then we have a CI/CD automated build and run of CloudSploit ready to go
-
In a Google Cloud Foundations account the only way to make changes to org policies is with Terraform service accounts. The only service that has access to this account is Cloud Build. In order to update Org Policies I created some Terraform which could be executed and run by Cloud Build in the customer’s account.
Just Do It.^TM^
You can do this today, right now. 120 build minutes per day are free with Google Cloud. There’s no reason not to! What problems do you have to solve? Can they be solved in an automated fashion with Cloud Build, DevOps practices and CI/CD?
Example Repo
https://github.com/EmileHofsink/cloudbuild-demo
References: