This article originally appeared on JetBrains on 4/25/19
Posted on April 25, 2019 by Yegor Naumov
This guest post is brought to you by Meni Besso, Product Manager at Quali, the creators of CloudShell Colony
CloudShell Colony is an Environment as a Service platform. It connects to your cloud providers including AWS, Azure, and Kubernetes and automates environment provisioning and deployment throughout your release pipeline in a scalable, maintainable, and smart way.
Creating a scalable CI/CD pipeline is harder than it looks.
Too many times have we witnessed DevOps engineers dive straight into configuring builds and writing scripts, only to find out later that they need to redo a significant part of their work. Any professional engineer will tell you that avoiding this unsavory outcome is easy. You need to treat your pipeline as a product: gather your requirements, design your architecture, develop, test, and improve your pipeline as your product grows.
In order to come up with a good design, you have to take three important aspects into account:
- Release workflow: Working out the steps that need to happen from the moment a developer commits code until the code goes live in production.
- Test automation: Planning the tests that need to run to validate the quality of the release and the automation tools that will be used to run them.
- Test environments: Determining the specifications for each test environment, taking into account test data, debugging, performance, and maintenance.
Much has been said about the first two items, but we at Quali believe that environments are the missing piece on the way to a reliable CI/CD pipeline, because continuous integration only works if you can trust your tests, and running tests on faulty environments is a serious barrier.
So, how do you design environments?
First, let’s distinguish between the two types of environment:
- Sandbox environments: Environments that you use temporarily to complete a specific task. For example, to run sanity tests or performance tests from your pipeline.
- Production environments: Environments that are always-on serving users. For example, your live production environment or a pre-production environment that you share with your customers.
A typical release pipeline uses both sandbox environments and production environments. You run various tests on sandboxes to validate your code and then deploy to production.
Sandbox environments can be tricky. Once you have set up your automated pipeline, you may notice the complexities. You need to support different variations of these environments, often in parallel. Some environments need to provide quick feedback while others need to replicate the full production environment. How do you deploy the artifacts, load test data, and set up your test tools? How do you debug failed tests? Are you using cloud infrastructure? How much does it cost?
At some point, you end up realizing that your challenge is also one of scale. Meaning, all these issues grow from bad to worse as your automation gradually consumes more and more environments.
Production environments can be tricky as well, but for different reasons. With production, you need to think about ongoing upgrades, handling downtime, backward compatibility, rollbacks, monitoring, and security.
Integrating your TeamCity CI/CD pipeline with an Environment as a Service platform like CloudShell Colony will ease many of these complexities and provides you with a more stable, reliable, and scalable CI/CD pipeline.
Let’s see how it works.
Integrating CloudShell Colony with your DevOps Toolchain
You start by creating a CloudShell Colony account and connecting it to your AWS or Azure accounts. This is done through a simple process that gives CloudShell Colony permission to create infrastructure in your public cloud accounts.
Then, download the installation instructions.
Once you have your TeamCity server connected to your CloudShell Colony account, you can connect your artifacts’ storage provider and start using CloudShell Colony in your release pipeline: run automated tests on its isolated sandbox environments and update your production using blue-green deployment.
Pipeline TeamCity and CloudShell Colony
Running Automated Tests on Sandbox Environments
To use sandbox environments in your TeamCity builds, you need to update your build configuration flow.
The first build step should start a sandbox. Use the ‘Colony’ runner, that comes out of the box with CloudShell Colony’s plugin, and fill in the required parameters. For example, enter the name of the environment blueprint you want to use, the location of your artifacts in the storage provider, and any test data you may wish to load to the sandbox environment.
When the environment is ready, provisioned and deployed, CloudShell Colony saves its details to a TeamCity variable, so you can extract any needed information (like IP addresses or application links) and feed your tests.
Finally, after the tests complete and the sandbox environment is no longer needed, you can terminate it by adding another build step that ends the sandbox and reclaims its cloud resources.
Creating Environments from Blueprints
CloudShell Colony’s environments are created from blueprints.
Blueprints define application requirements in a simple YAML format that you store in your source control provider. They include the definitions of artifacts, networking and computing requirements, deployment scripts, variables, and test data. They can also integrate with traditional IaC (Infrastructure-as-Code) tools, such as Terraform.
Once you create your blueprints and store them in your source control repository, they will be available in CloudShell Colony’s self-service catalog, allowing you to launch environments from CloudShell Colony’s user interface or from your TeamCity builds.
Environment Lifecycle
Every environment goes through several stages including a verification step that ensures that the environment is 100% ready for your automated tests.
And since your environment is composed of applications, CloudShell Colony manages the application dependencies and initialization and provides quick debugging and troubleshooting options.
In case of errors, you will know exactly which steps failed and you are able to browse the deployment logs and securely connect to your compute instances.
When the environment is no longer needed, CloudShell Colony cleans up the environment’s infrastructure from your cloud account, ensuring that you won’t have to pay for unused infrastructure.
Environments have a crucial part in building a stable application pipeline, and in DevOps processes in general. Using Platforms like CloudShell Colony enables teams to get fully provisioned and deployed environments quickly while keeping cloud expenses under control.
Like what you’ve just read? Start your free trial of CloudShell Colony.
