Since Open Infrastructure Summit in Denver earlier this year I've spent some time ramping up on Kubernetes Operators and Golang. Perhaps “some time” is an understatement, this is more of a coaxed deep dive into some technology which I knew existed but hadn’t paid proper attention too yet. (hint: I was asked to look into this). In the past few weeks I've created a working version of a Keystone Kubernetes Operator in Go.

The Keystone Operator is based on the operator-sdk which provides a nice way to cookie cut the Operator application structure. It also provides some nice benefits for releasing and installing the Operator in your Kubernetes cluster. There are several ways to install an Operator which make them really flexible.

For local development

If you've built your k8s Operator with the operator-sdk you've got access to some niceties and one is the 'up local' command. Below is an example of how to use this with the current Keystone CRD:

kubectl create -f [deploy/crds/keystone_v1_keystoneapi_crd.yaml]
operator-sdk up local

These commands create the required CRD and then launch a working version of the k8s Operator. The last command there stays running in your local terminal and logs any output from your k8s Operator. As long as you leave it running you can run API commands to create k8s custom resources provided by your Operator. It’s important to note that by default this command runs with your local credentials and the Operator has access to any permissions that you have as well. There are some options to fine tune 'up local' though so it runs with different creds (--kubeconfig) or in an alternate namespace (--namespace) if you need them too. See (operator-sdk --help for more)

Deploying an Operator on the CLI

Because Operators are containers that are deployed in your Kubernetes cluster, deploying them is building and publishing a container. The operator-sdk has a few commands that make this streamlined:

operator-sdk build openstack-keystone --image-builder buildah
podman tag <image id> quay.io/openstack-k8s-operators/keystone-operator:devel
podman push quay.io/openstack-k8s-operators/keystone-operator:devel

Once you've uploaded your container to a registry you can deploy it into your cluster on the CLI using files in the 'deploy' directory. Using either kubectl or oc you can deploy them like this:

kubectl create -f deploy/service_account.yaml
kubectl create -f deploy/role.yaml
kubectl create -f deploy/role_binding.yaml
kubectl create -f deploy/operator.yaml

Deploying an Operator via OLM

Lastly there is a cool new Operator Lifecycle Manager that takes care of some of the k8s Operator management for you. I won't get into all the details here but the benefits of using the OLM are automatic k8s Operator deployment from channels, dependency management between operators (a bit crude at the moment but promising), and integration with OperatorHub.

In order to integrate with OLM you need a CSV (clusterserviceversion) and the operator-sdk makes this fairly easy too via the 'operator-sdk olm-catalog gen-csv' command. When you initially create a CSV some things are automatically generated but a few others need to be manually specified. Say you've already crafted a 0.0.1 CSV and you want to release 0.0.2 you could do it like this via the operator-sdk.

operator-sdk olm-catalog gen-csv --csv-channel default --operator-name keystone-operator --csv-version 0.0.2 --verbose --from-version 0.0.1

Generated CSV's can be deployed on the CLI just like any other custom resource.

kubectl create -f deploy/olm-catalog/keystone-operator/0.0.1/keystone-operator.v0.0.1.clusterserviceversion.yaml

This will take care of checking (and possibly installing) dependencies for roles, service accounts, and ultimately if everything checks out it will fire off a Deployment to install the k8s Operator in the cluster. It even has a spot for a nice icon there which will show up in your k8s console if you've got the OLM components installed there.

Summary

As someone who has worked on OpenStack deployment for the last few years, k8s Operators check a lot of the boxes that I'm passionate about. Assuming your goal is to deploy on k8s, when using k8s Operators your deployment tools are built using the same framework as your production applications (i.e k8s API's). The tools have an API that is provided as a CRD just like any other k8s resources you are using and can be managed as such. Cutting a new release of the deployment tools is as simple as building and publishing a container. Lastly, given the tools above the integration with the dev/test process is very streamlined. I think I could get used to this.