What is helm?
Helm is a tool for deploying and maintaining Kubernetes resources.
It does this through three key features:
-
String templating
-
Version Control
-
Ownership
Templates
Helm uses a variation of GO language to take a values file and apply it to a series of templates. This separation allows for site-to-site values to be separated out from structure and logic. Helm understands what all the kubernetes resources are and deploys them in order of dependencies (e.g. namespaces are deployed before resources that use them).
Version Control & Ownership
When helm creates and deploys resources (deployments/pods/config maps/etc) it tags it with some special information relating back to the helm chart and version. This allows helm to know what resources have been deployed by each chart and if things need to be wound back, this can also be controlled via helm.
How is the Youtap Chart structured?
Subcharts
See https://helm.sh/docs/chart_template_guide/subcharts_and_globals/ for information about subcharts.
Putting all of the services, configmaps, ingresses, and other templates in a single chart means that it gets very complicated when configuring everything the entire cluster needs, especially in the values.yaml file. This also means that the versioning (when it is updated) doesn’t mean much from one number to the next. Splitting the chart into subcharts, with the parent chart handling mainly the versioning of the subcharts and definitions of common templates, means we can make each one into a more manageable size.
The subcharts being smaller and updated less frequently means that if we stay on top of the proper versioning, then we can track the history of the charts more easily, and potentially store the chart in a repository for later reference.
Using Loops
The traditional structure for helm is to have a series of values that are referenced by template files.
Considering the number of microservices and the desire for quick changes, youtap currently uses lists and maps in the values file to reuse templates.
The tradeoff for this strategy is complexity. The templates can often get complicated trying to contain the different types of configurations and options required for lost of different use cases.
Examples
In the values file there is a list of namespaces
namespaces:
- name: api-gateway
- name: approval-rest-api
- name: audit-rest-api
- name: auth-server
- name: checkout-rest-api
- name: configuration-server
- name: customer-server
...
and a simple template:
{{- $top := . -}}
{{- range $namespace := .Values.namespaces -}}
---
apiVersion: v1
kind: Namespace
metadata:
name: {{ $namespace.name }}
---
{{ end }}
This template uses the {{- range $namespace := .Values.namespaces -}} command to loop through the list and produce the required resources:
We can see the result using helm template .
---
# Source: helm-application-deployment/templates/namespaces.yaml
apiVersion: v1
kind: Namespace
metadata:
name: api-gateway
---
# Source: helm-application-deployment/templates/namespaces.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: approval-rest-api
---
# Source: helm-application-deployment/templates/namespaces.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: audit-rest-api
---
# Source: helm-application-deployment/templates/namespaces.yaml
---
...
This list structure is reused through many of the templates.
Other templates use maps, which function in much the same way:
{{- $top := . -}}
{{- range $name, $configmap := .Values.configmaps -}}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: {{$name}}
namespace: {{$configmap.namespace}}
data:
{{ toYaml $top.Values.defaults.configmap.data | nindent 2 }}
{{- if not (empty $configmap.configFile) }}
application.yml: |-
{{- tpl ($.Files.Get $configmap.configFile) . | nindent 4 }}
{{- end }}
{{- if not (empty $configmap.data) }}
{{ toYaml $configmap.data | nindent 2 }}
{{- end }}
---
{{ end }}
Thi s command {{- range $name, $configmap := .Values.configmaps -}} loops through config maps as well but also returns the key with $name. This allows a list with keys that can be used in the template:
configmaps:
yts-portal-config:
namespace: yts-portal
fsp-rest-api-config:
namespace: fsp-rest-api
data:
LOGGING_LEVEL_COM_YOUTAP: TRACE
FEIGN_CLIENT_CONFIG_DEFAULT_LOGGERLEVEL: FULL
Templates can be used as normal without lists but this structure allows new resources to be deployed by simply appending the list/map in the deployment file.
Configuration Injection
Conventionally, the config maps have been populated from the helm chart itself, and the pods have fallen back on default yml configuration within the image as a default. However, in an effort to make the configuration updates more streamlined, and to help ease the transition to multi-tenancy, we have been investigating how we might load configuration from a repository that can be updated by an operations staff member with limited/no knowledge of Helm.
YML File Injection
Whole YML files can be used as configuration in addition to the config map values. To do so, add a file into the config directory in the base directory of the project. Then, in the configmap entry for the service add a “configMap” value that is the name of the file (including “config/”. For example:
configmaps:
api-gateway-config:
namespace: api-gateway
configFile: "config/api-gateway.yml"
Repository-Backed Configuration Service and Pod Environment Refresh (without restarting)
Spring Cloud Config allows a server to be configured for central configuration management. John has already implemented a version of this server. Services with the Spring Cloud Config dependency can read from the server when configured properly.
https://docs.spring.io/spring-cloud-config/docs/current/reference/html/#_quick_start
Actuator-enabled pods can have their configuration updated without restarting. There is an endpoint (see “How Spring Boot Reloads…” that triggers an environment update on all pods for a service through a bus via Rabbit/Kafka, which would be useful when we scale up.
The general concept is that when the pods start, they get their configuration from the Spring Cloud Config Server. The configuration itself is stored in a Bitbucket repository, which the cloud config service has permission to read from. When a user updates some of the configuration that will trigger a pipeline that sends a message to one or more of the actuator endpoints that triggers the services to refresh their environments and recreate the required beans without an interruption in the service.
Overloading
A key concept used in the chart structure if overloading.
At the top of the values file a series of default values are set for the image repository, config map values (db url, password, etc), annotations and resources.
These are used by default in templates but the option is then given to overload on a resource to resouce basis.
Examples
Default resource for a deployment
resources:
requests:
memory: "128Mi"
cpu: "50m"
limits:
memory: "512Mi"
cpu: "330m"
Then overloaded on fineract deployment with more memory and cpu as required
deployments:
- name: fineract
namespace: fineract
port: 443
labels:
app.kubernetes.io/name: fineract
configmap: fineract-config
resources:
limits:
cpu: "1000m"
memory: "1Gi"
requests:
cpu: "200m"
memory: "0.5Gi"
Differences with Development Pipelines
The gitlab pipelines that deploy to youtap-development do not use helm. Manifest files are maintained per microservice and pushed directly using kubectl apply -f. This allows for quick testing of changes but is difficult to maintain for upgrades to sites (the manifest file would likely be 1000s of lines!).
Further reading: https://helm.sh/docs/topics/charts/