"I need warp speed now!"

Just another day in Engineering

What is (not) an operator?

Not a product...an API...or a library.

An operator is a pattern

A resident, privileged container...

...which watches for CRDs, and performs actions

Custom Resource Definition (CRD)

Custom Resource Definition (CRD)

Custom Resource Definition (CRD)

Model the structure needed for you app

Not what's needed to run it on k8s

Operator SDK

A framework to create operators

Open source, sponsored by Red Hat

Getting the SDK

Homebrew (macOS), Github release

Getting the SDK

Homebrew (macOS), Github release

sdk.operatorframework.io/docs/installation

Other dependencies

kubectl, Docker Desktop

Go-lang

Helm

Ansible, Ansible runner, Openshift Python

Power up the replicator

Creating a project

Initialization

Domain

Not (necessarily) a web domain

Namespace for API definitions (kinds)

Choosing a domain

Shorter is better

So how do I choose?

domain is your org

group is your app

So how do I choose?

domain is your org

group is your app

Plugins

The runtime env for your operator

Go, Helm, Ansible

Go lang

Most flexible, but most difficult

Best for new, from-scratch operators

Helm

Easiest, but least flexible

Add CRD support to existing chart

Ansible

More flexible than Helm,

less complex than Go

The middle ground

Interpreted, not compiled

Built-in modules for k8s, DBs, Cloud, and more

The middle ground

Interpreted, not compiled

Built-in modules for k8s, DBs, Cloud, and more

When to use

Port existing CI scripts to CRDs

Simpler apps which need stateful changes

init in subdirectory

Aids in build automation

Directory layout

path/to/my-operator/src
  ├── config/
  ├── Dockerfile
  ├── Makefile
  ├── molecule/
  ├── playbooks/
  ├── PROJECT
  ├── requirements.yml
  ├── roles/
  └── watches.yaml

Directory layout

path/to/my-operator/src
  ├── config/                    <-- CRDs, Schema, Samples, etc.
  ├── Dockerfile
  ├── Makefile
  ├── molecule/
  ├── playbooks/
  ├── PROJECT
  ├── requirements.yml
  ├── roles/
  └── watches.yaml

Directory layout

path/to/my-operator/src
  ├── config/
  ├── Dockerfile                 <-- Operator Dockerfile
  ├── Makefile
  ├── molecule/
  ├── playbooks/
  ├── PROJECT
  ├── requirements.yml
  ├── roles/
  └── watches.yaml

Directory layout

path/to/my-operator/src
  ├── config/
  ├── Dockerfile                 Ansible files
  ├── Makefile                   -------------
  ├── molecule/                  <-- tests
  ├── playbooks/                 <-- source
  ├── PROJECT
  ├── requirements.yml           <-- dependencies
  ├── roles/                     <-- more source
  └── watches.yaml

Directory layout

path/to/my-operator/src
  ├── config/
  ├── Dockerfile
  ├── Makefile
  ├── molecule/
  ├── playbooks/
  ├── PROJECT
  ├── requirements.yml
  ├── roles/
  └── watches.yaml               <-- SDK configuration

Gold, Blue or Red?

Creating an API

Creating a kind

What does it do?

1. Creates a CRD template
2. Updates watches.yaml
3. Creates Ansible Role *

* --generate-role only

CRD Schema

One for each API

config/crd/bases/

Structure, Fields, Types

Based on OpenAPI v3 schema

"Accept all" schema by default

Structure, Fields, Types

Based on OpenAPI v3 schema

"Accept all" schema by default

Structure, Fields, Types

Based on OpenAPI v3 schema

"Accept all" schema by default

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: mysqlclusters.myapp.myorg.tld
spec:
  group: myapp.myorg.tld
  names:
    kind: MySQLCluster
    listKind: MySQLClusterList
    plural: mysqlclusters
    singular: mysqlcluster
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: MySQLCluster is the Schema for the mysqlclusters API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: Spec defines the desired state of MySQLCluster
            type: object
            x-kubernetes-preserve-unknown-fields: true
          status:
            description: Status defines the observed state of MySQLCluster
            type: object
            x-kubernetes-preserve-unknown-fields: true
        type: object
    served: true
    storage: true
    subresources:
      status: {}

Self-sealing stem bolts

Writing the role

Cluster access

Operator relies on service account, role

path/to/my-operator/src
└── config
    └── crd
        ├── role_binding.yaml
        ├── role.yaml
        └── service_account.yaml

Default permissions

Work with your CRDs,

and common k8s defs...but not configmaps.

Default permissions

Work with your CRDs,

and common k8s defs...but not configmaps.

Default permissions

Work with your CRDs,

and common k8s defs...but not configmaps.

Input

CRD defintion created, deleted, edited

Passed as Ansible vars

Metadata values

{{ ansible_operator_meta.name }}
{{ ansible_operator_meta.namespace }}

Differs in older SDK versions

Spec values

Contents exported as top-level Ansible vars

CamelCase → snake_case

Sample CRD

apiVersion: myapp.myorg.tld/v1
kind: MySQLCluster
metadata:
  name: mysqlcluster-sample
spec:
  mysqlAdminSecret: mysql-admin      #
  mysqlReadersSecret: mysql-reader   #  Note the CamelCase
  size: "20Gi"                       #

Vars in Ansible

---
- name: Get the spec variables
  debug:
    msg: |
        {{ ansible_operator_meta.name }}
        {{ ansible_operator_meta.namespace }}
        {{ mysql_admin_secret }}              #
        {{ mysql_readers_secret }}            #  snake_case here
        {{ size }}                            #

Creating, updating defs

k8s module

Local cluster already authenticated

---
- name: Create mysql writer service
  k8s:
    definition: |
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: {{ ansible_operator_meta.name }}
        labels:
          app.kubernetes.io/managed-by: "my-operator"
          app.kubernetes.io/created-by: "my-manager"
      spec:
        clusterIP: None
        ports:
          - name: mysql
            port: {{ mysql_port | default('3306') }}
            protocol: TCP
        selector:
          app.kubernetes.io/name: "mysql"
          mysql-cluster: "{{ ansible_operator_meta.name }}"
    namespace: "{{ ansible_operator_meta.namespace }}"

(Scroll down)

Deployment

1. Build the docker image
2. Push the image to registry
3. Push defitions to cluster

Configure image and tag

Set in Makefile

IMG ?= https://registry.example.com/my-operator:x.y.z

Configure image and tag

Set in Makefile

IMG ?= https://registry.example.com/my-operator:x.y.z

Build the docker image

make docker-build

Builds and tags the image

Push to registry

docker login

make docker-push

Apply defs

make deploy

Applies to default authorized cluster

Deploy a CRD

YAML in config/samples/

Edit, then kubectl apply

Deployment run

Creates new k8s objects

Deployments, StatefulSets, Pods, etc.

Deployment run

Creates new k8s objects

Deployments, StatefulSets, Pods, etc.

Deployment run

Creates new k8s objects

Deployments, StatefulSets, Pods, etc.

Reconcile run

Validates current config against operator

Must not have any changed

Reconcile run

Validates current config against operator

Must not have any changed

Viewing results

kubectl edit the CRD

status.ansibleResult section

apiVersion: flightdeck.t7.io/v1
kind: MySQLCluster
...
status:
  conditions:
  - ansibleResult:
      changed: 0
      completion: 2021-07-19T15:57:27.258505
      failures: 1
      ok: 9
      skipped: 6
    lastTransitionTime: "2021-07-19T15:57:27Z"
    message: unknown playbook failure
    reason: Failed
    status: "False"
    type: Failure

(Scroll down)

Tailing the manager

kubectl logs -f my-operator-pod -c manager

Useful for debugging

Tailing the manager

kubectl logs -f my-operator-pod -c manager

Useful for debugging

--------------------------- Ansible Task StdOut -------------------------------

 TASK [Work with the writer] ********************************
fatal: [localhost]: FAILED! => {"reason": "Could not find or access '/tmp/ansible-operator/runner/mykind.myapp.tld/v1/MySQLCluster/my-namespace/mysqlcluster-sample/project/writer.yml' on the Ansible Controller."}

-------------------------------------------------------------------------------

(Scroll right)

Clean up

make undeploy

Mind your finalizers!

Computer, Arch

Stateful changes

Any non-k8s resource

On-cluster applications

External APIs

No reconcile run

No k8s defs created, so not needed

Looking up defs

k8s lookup

Local cluster already authenticated

- name: Get CRD
  set_fact:
    _crd: "{{ lookup('k8s', kind='my-kind', namespace='my-namespace', resource_name=_var_with_name)}}"

(Scroll right)

Where to use

Inline

Task-level vars

set_fact task

Failed lookups

Returns empty list, does not fail the run!

(_def_lookup | default([])) | length < 1

Fail module

Causes run to fail

Set message to aid debugging

Fail module

Causes run to fail

Set message to aid debugging

- name: Fetch the MySQLCluster definition
  set_fact:
    _cluster_def: "{{ lookup('k8s', kind='MySQLCluster', namespace=_cluster_def_namespace, resource_name=cluster.name) }}"
  vars:
    _cluster_def_namespace: "{{ cluster.namespace | default(ansible_operator_meta.namespace) }}"
- fail:
    msg: "Could not find requested MySQLCluster {{ cluster.name }}"
  when:
    - "(_cluster_def | default([])) | length < 1"

(Scroll right)

Stateful deletions

Must be handled by your operator

No auto-cleanup like k8s defs

Finalizers

Run when deleting a k8s def

Specified in watches.yaml

Ansible Finalizers

Existing role, playbook with custom vars

New role or playbook

Role recommended

Seperate from creation role

My result in some duplication, but easier!

Creating the role

Init manually, name after creation role

ansible-galaxy init path/to/my-operator/src/roles myRole_finalizer

Update watches.yaml

Add finalizer key,

point to new role.

---
- version: v1
  group: myapp.myorg.tld  #
  kind: MySQLCluster      #  From  `operator-sdk create api`
  role: mysqlcluster      #
  finalizer:
    name: myapp.myorg.tld/cluster-finalizer
    role: mysqlcluster_finalizer

---
- version: v1
  group: myapp.myorg.tld
  kind: MySQLCluster
  role: mysqlcluster
  finalizer:                                  #
    name: myapp.myorg.tld/cluster-finalizer   # Created manually
    role: mysqlcluster_finalizer              #

Finalizers and undeploy

make undeploy kills operator first

Delete defs manually, or finalizers: []

Modifying the image

Update Dockerfile

Based on RedHat UBI

Adding libraries

pip for Python-only libs

yum for everything else

Thanks Jeff Geerling!

FROM quay.io/operator-framework/ansible-operator:latest

COPY requirements.yml ${HOME}/requirements.yml
RUN ansible-galaxy collection install \
 -r ${HOME}/requirements.yml \
 && chmod -R ug+rwx ${HOME}/.ansible

COPY watches.yaml ${HOME}/watches.yaml
COPY roles/ ${HOME}/roles/
COPY playbooks/ ${HOME}/playbooks/

USER root

ADD https://dev.mysql.com/get/mysql80-community-release-el8-1.noarch.rpm /tmp/mysql80-community-release-el8-1.noarch.rpm

RUN yum install -y /tmp/mysql80-community-release-el8-1.noarch.rpm && \
    yum install -y mysql-community-client mysql-community-common mysql-community-libs python38-PyMySQL

USER ansible

FROM quay.io/operator-framework/ansible-operator:latest

COPY requirements.yml ${HOME}/requirements.yml   #
RUN ansible-galaxy collection install \          #
 -r ${HOME}/requirements.yml \                   #
 && chmod -R ug+rwx ${HOME}/.ansible             #
                                                 #  Lines from SDK
COPY watches.yaml ${HOME}/watches.yaml           #
COPY roles/ ${HOME}/roles/                       #
COPY playbooks/ ${HOME}/playbooks/               #

USER root

ADD https://dev.mysql.com/get/mysql80-community-release-el8-1.noarch.rpm /tmp/mysql80-community-release-el8-1.noarch.rpm

RUN yum install -y /tmp/mysql80-community-release-el8-1.noarch.rpm && \
    yum install -y mysql-community-client mysql-community-common mysql-community-libs python38-PyMySQL

USER ansible

FROM quay.io/operator-framework/ansible-operator:latest

COPY requirements.yml ${HOME}/requirements.yml
RUN ansible-galaxy collection install \
 -r ${HOME}/requirements.yml \
 && chmod -R ug+rwx ${HOME}/.ansible

COPY watches.yaml ${HOME}/watches.yaml
COPY roles/ ${HOME}/roles/
COPY playbooks/ ${HOME}/playbooks/

USER root     # change to root to run installs

ADD https://dev.mysql.com/get/mysql80-community-release-el8-1.noarch.rpm /tmp/mysql80-community-release-el8-1.noarch.rpm

RUN yum install -y /tmp/mysql80-community-release-el8-1.noarch.rpm && \
    yum install -y mysql-community-client mysql-community-common mysql-community-libs python38-PyMySQL

USER ansible

FROM quay.io/operator-framework/ansible-operator:latest

COPY requirements.yml ${HOME}/requirements.yml
RUN ansible-galaxy collection install \
 -r ${HOME}/requirements.yml \
 && chmod -R ug+rwx ${HOME}/.ansible

COPY watches.yaml ${HOME}/watches.yaml
COPY roles/ ${HOME}/roles/
COPY playbooks/ ${HOME}/playbooks/

USER root
# Download MySQL
ADD https://dev.mysql.com/get/mysql80-community-release-el8-1.noarch.rpm /tmp/mysql80-community-release-el8-1.noarch.rpm

RUN yum install -y /tmp/mysql80-community-release-el8-1.noarch.rpm && \
    yum install -y mysql-community-client mysql-community-common mysql-community-libs python38-PyMySQL

USER ansible

FROM quay.io/operator-framework/ansible-operator:latest

COPY requirements.yml ${HOME}/requirements.yml
RUN ansible-galaxy collection install \
 -r ${HOME}/requirements.yml \
 && chmod -R ug+rwx ${HOME}/.ansible

COPY watches.yaml ${HOME}/watches.yaml
COPY roles/ ${HOME}/roles/
COPY playbooks/ ${HOME}/playbooks/

USER root

ADD https://dev.mysql.com/get/mysql80-community-release-el8-1.noarch.rpm /tmp/mysql80-community-release-el8-1.noarch.rpm
# Install MySQL client and Python libs
RUN yum install -y /tmp/mysql80-community-release-el8-1.noarch.rpm && \
    yum install -y mysql-community-client mysql-community-common mysql-community-libs python38-PyMySQL
# Switch back to ansible user.
USER ansible

(Scroll down)

Activate the Emergency Engineering Hologram!

Automating deployment

Directory layout

src/ helps here!

Dockerfile

Nothing fancy here

Public base image, no complex build

Build like any container

docker build, tag, and push

Docker Hub automated build

Replacing make deploy

Conventional manifest

Helm chart

Getting the manifest

Capture kustomize output

Hack the makefile!

deploy: kustomize
  cd config/manager && $(KUSTOMIZE) edit set image controller=${IMG}
  $(KUSTOMIZE) build config/default | kubectl apply -f -

to

deploy: kustomize
  cd config/manager && $(KUSTOMIZE) edit set image controller=${IMG}
  $(KUSTOMIZE) build config/default > path/to/output.yml -

Helm chart

Often used to dpeloy operators

Templating, versioning, 3rd party integrations

Creating the chart

Use kustomize output

Break up into files, add helm labels

Directory layout

path/to/my-operator
└── charts
    └── my-chart
        ├── Chart.yaml
        ├── templates
        │   ├── crds.yaml
        │   ├── deployment.yaml
        │   ├── _helpers.tpl
        │   └── rbac.yaml
        └── values.yaml

(You'll see why in a sec!)

Distribution

Need a Helm repository

Hosts chart tarballs by version

Github as a helm repo

Pages hosts the repo

Actions builds the chart

Configuring pages

gh-pages branch

Branch must be empty

Set up Action

Uses stefanprodan/helm-gh-pages

Commit to .github/workflows/release-chart.yml

Override the version

Sets app, chart version to gh-pages tag

Ignores version in Chart.yaml

name: release-chart
on:
  push:
    tags: '*'

jobs:
  release-chart:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Set env
        run: echo "RELEASE_VERSION=${GITHUB_REF#refs/*/}" >> $GITHUB_ENV
      - name: Publish Helm chart
        uses: stefanprodan/helm-gh-pages@master
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          charts_dir: charts
          branch: gh-pages
          app_version: ${{ env.RELEASE_VERSION }}
          chart_version: ${{ env.RELEASE_VERSION }}

(Scroll down.)

Deployment

Commit to gh-pages, tag, push

Hub builds image

Action builds Chart

Further reading

Introduction to Operators

Operator SDK Ansible Quickstart

Helm chart getting started

Github as a Helm repo

A 3D printed, Raspberry Pi-powered Tricorder

Special thanks

This event!

TEN7

patreon.com/socketwench

Thank you!

socketwench.github.io/engineering-i-need-more-power