Kubernetes deployment and configuration management tools
Many companies are moving towards Kubernetes for handling their container workloads. To learn more about Kubernetes, look at this video course. Although Kubernetes has become quite stable over the years, deployment to Kubernetes is still not evolved as an art. This is where Kubernetes kustomize comes in. You have probably heard about helm but there are a few tools out there that people have been using for deployment. Here are a few
- Helm – Helm is a package manager that provides GO based templating framework for the YAML files. Helm has a concept of charts where each application is packaged as a chart and you can use this chart to deploy an application. The disadvantage with Helm, however, is that you need to install a server component called Tiller and also Helm charts might get complicated especially when you need to maintain multiple environments.
- Kapitan – Kapitan is another template based configuration management tool for Kubernetes. You need to create jsonnet files to create the YAML file and use jinja2 based templating. Kapitan has a high learning curve, since you need to learn jsonnet and jinja2.
- kubecfg : kubecfg is another jsonnet based configuration tool for Kubernetes deployment.
- This google spreadsheet lists lot of other tools that are part of the ecosystem
- Kustomize – We will talk about Kustomize in this article.
Need for Kubernetes Kustomize and other tools
Before we look at Kustomize, lets understand what we need for a production Kubernetes deployment
- Maintaining the deployment code in a repository. All the code should be maintained in a repository and we should be able to maintain audit trails of changes
- Ability to manage multiple environments. Its not uncommon for companies to have around 5 to 6 different environments. We don’t want to replicate the same code for all environments
- Ability for each environment to be in its own repository. I would be good if the files for the different environments can be separated into different projects so that a user has only access to a particular environment.
Kustomize allows us to do all of the above. lets look at Kustomize now.
What is Kubernetes Kustomize
Kustomize is a project part of the Kubernetes SIGs and it can be used to specify configuration for Kubernetes deployment in a very simplified way. It does not use templates but still allows separating out configuration for the various environments and maintaining them as code. Kustomize allows maintaining a base set of files and a special file called kustomization.yaml that combines the base files and applies common logic to them to create a single deployment manifest. The common logic could be adding labels or create configuration.
Kubernetes Kustomize basic example
It will be easier to explain that with an example. The source code for this example is at
To install Kustomize on linux download the binary from the releases page. Once the binary has been downloaded rename it to kustomize and put it in a folder that is in application path (e.g. /usr/local/bin/)
Kubernetes kustomize nginx deployment
Here’s a folder structure for our deployment
The files nginx-deployment.yaml contains deployment configuration for the nginx server. Here’s how it looks.
nginx-service.yaml creates the service. Here’s how it looks
The configuration file for nginx is inside the config folder and is called nginx.conf. The contents of the file are not important for this tutorial, so we will ignore it. The most important file for our understanding currently is kustomization.yaml. This file does the following things:
- Combines all the yaml files that are required for this installation.
- Adds additional values such as labels and annotations.
- Creates the configmap and secrets
Here’s how the file looks:
The ‘resources’ section tells which files are included in the final manifest. namespace: dev is added to each object in the files included in the resources section. namePrefix adds the prefix ‘dev’ to each name. commonLabels adds the label ‘owner: teamA’ to each object. Similarly CommonAnnotations adds an annotation to each object. ConfigMapGenerator is used to generate a configMap. In this case we are generating a configMap called nginx.conf from the config/nginx.conf file and app.home that contains the variable APP_HOME=/opt/app.
The next step is to now use the kustomize binary to create the manifest. To build the manifest, cd to the folder containing the kustomization.yaml and other files and type in
kustomize build .
This will combine the files, add the additional values as specified in kustomization.yaml, create the configMaps and then generate a combined file.
Here’s part of the output
A single command to apply the output to Kubernetes is
kustomize build . | kubectl apply -f -
This finishes the first article in the Kubernetes kustomize series. In the second article we will look at how to use kustomize to create multiple environments.