The previous post explained what a Kubernetes Operator is: a controller that watches a custom resource and continuously reconciles the actual cluster state toward the desired state. That explanation is useful. This one is different. This one ends with a working Operator.
We're going to build one end-to-end: define a custom resource, write the reconciliation logic in Go, see the equivalent in Java, wire up the RBAC, and deploy it to a cluster. The example is intentionally simple, so the scaffolding and the pattern are visible rather than buried under domain complexity.
What we're building
Our Operator will manage a custom resource called WebApp. When a user creates a WebApp, the Operator creates a Deployment and a Service on their behalf. When the user updates the WebApp spec, the Operator updates those child resources accordingly. When the WebApp is deleted, the child resources are cleaned up automatically.
This is the "hello world" of Operators: it encodes no deep application knowledge, but it exercises every part of the pattern. Every Operator you ever build, regardless of complexity, goes through the same steps.
The WebApp spec will have three fields: the container image to run, the number of replicas, and the port to expose. A user will create resources like this:
apiVersion: apps.codelooru.com/v1alpha1
kind: WebApp
metadata:
name: my-frontend
namespace: default
spec:
image: nginx:1.25
replicas: 3
port: 8080The Operator sees this, creates a three-replica Deployment and a ClusterIP Service, and keeps them in sync with whatever the user sets in the spec.
Scaffolding the project
Kubebuilder is the official scaffolding tool from the Kubernetes project. It generates the project layout, the CRD registration code, the controller skeleton, and the Makefile targets to install and run everything. Install it, then run:
kubebuilder init --domain codelooru.com --repo github.com/codelooru/webapp-operator
kubebuilder create api --group apps --version v1alpha1 --kind WebAppAnswer y to both prompts (create resource, create controller). Kubebuilder generates a directory structure like this:
webapp-operator/
api/
v1alpha1/
webapp_types.go ← your CRD struct lives here
groupversion_info.go
internal/
controller/
webapp_controller.go ← your reconciler lives here
config/
crd/ ← generated CRD YAML
rbac/ ← generated RBAC YAML
manager/ ← Deployment YAML for the Operator pod
cmd/
main.go ← wires everything togetherYou never edit the config/ directory by hand. Kubebuilder regenerates it from your Go types and annotations when you run make generate manifests.
Defining the CRD
In Kubebuilder, the CRD schema is derived from a Go struct. Open api/v1alpha1/webapp_types.go and define the spec and status:
// WebAppSpec defines the desired state of WebApp.
type WebAppSpec struct {
// +kubebuilder:validation:Required
// +kubebuilder:validation:MinLength=1
Image string `json:"image"`
// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:Maximum=20
// +kubebuilder:default=1
Replicas int32 `json:"replicas,omitempty"`
// +kubebuilder:validation:Minimum=1
// +kubebuilder:validation:Maximum=65535
Port int32 `json:"port"`
}
// WebAppStatus defines the observed state of WebApp.
type WebAppStatus struct {
// +listType=map
// +listMapKey=type
Conditions []metav1.Condition `json:"conditions,omitempty"`
ReadyReplicas int32 `json:"readyReplicas,omitempty"`
}
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Image",type=string,JSONPath=`.spec.image`
// +kubebuilder:printcolumn:name="Replicas",type=integer,JSONPath=`.spec.replicas`
// +kubebuilder:printcolumn:name="Ready",type=integer,JSONPath=`.status.readyReplicas`
type WebApp struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec WebAppSpec `json:"spec,omitempty"`
Status WebAppStatus `json:"status,omitempty"`
}The comment markers starting with // +kubebuilder: are not decorative. They are annotations that the code generator reads. +kubebuilder:validation:Minimum=1 becomes a validation rule in the generated CRD YAML. +kubebuilder:printcolumn adds columns to kubectl get webapp output. +kubebuilder:subresource:status creates a separate status subresource so that updating status doesn't conflict with spec updates.
Run make generate manifests and Kubebuilder produces the full CRD YAML in config/crd/bases/. You never write that YAML by hand.
Writing the reconciler
The reconciler is where the Operator's intelligence lives. Open internal/controller/webapp_controller.go. Kubebuilder has already generated the function signature; you fill in the body.
The full reconciler for WebApp looks like this:
// +kubebuilder:rbac:groups=apps.codelooru.com,resources=webapps,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=apps.codelooru.com,resources=webapps/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
func (r *WebAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
log := log.FromContext(ctx)
// 1. Fetch the WebApp resource.
webapp := &appsv1alpha1.WebApp{}
if err := r.Get(ctx, req.NamespacedName, webapp); err != nil {
// Not found means it was deleted. Nothing to do — owner references
// handle child resource cleanup automatically.
return ctrl.Result{}, client.IgnoreNotFound(err)
}
// 2. Reconcile the Deployment.
if err := r.reconcileDeployment(ctx, webapp); err != nil {
return ctrl.Result{}, err
}
// 3. Reconcile the Service.
if err := r.reconcileService(ctx, webapp); err != nil {
return ctrl.Result{}, err
}
// 4. Update status.
if err := r.updateStatus(ctx, webapp); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{}, nil
}This top-level function is deliberately short. Each resource type gets its own helper that handles the full create-or-update logic. Let's look at reconcileDeployment:
func (r *WebAppReconciler) reconcileDeployment(ctx context.Context, webapp *appsv1alpha1.WebApp) error {
desired := r.deploymentForWebApp(webapp)
// Set the WebApp as the owner of this Deployment.
// When the WebApp is deleted, the Deployment is garbage-collected automatically.
if err := ctrl.SetControllerReference(webapp, desired, r.Scheme); err != nil {
return err
}
existing := &appsv1.Deployment{}
err := r.Get(ctx, types.NamespacedName{Name: desired.Name, Namespace: desired.Namespace}, existing)
if errors.IsNotFound(err) {
// Deployment doesn't exist yet — create it.
return r.Create(ctx, desired)
}
if err != nil {
return err
}
// Deployment exists — patch it to match the desired state.
patch := client.MergeFrom(existing.DeepCopy())
existing.Spec.Replicas = desired.Spec.Replicas
existing.Spec.Template.Spec.Containers[0].Image = desired.Spec.Template.Spec.Containers[0].Image
return r.Patch(ctx, existing, patch)
}
func (r *WebAppReconciler) deploymentForWebApp(webapp *appsv1alpha1.WebApp) *appsv1.Deployment {
labels := map[string]string{"app": webapp.Name}
replicas := webapp.Spec.Replicas
return &appsv1.Deployment{
ObjectMeta: metav1.ObjectMeta{
Name: webapp.Name,
Namespace: webapp.Namespace,
},
Spec: appsv1.DeploymentSpec{
Replicas: &replicas,
Selector: &metav1.LabelSelector{MatchLabels: labels},
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{Labels: labels},
Spec: corev1.PodSpec{
Containers: []corev1.Container{{
Name: "app",
Image: webapp.Spec.Image,
Ports: []corev1.ContainerPort{{ContainerPort: webapp.Spec.Port}},
}},
},
},
},
}
}Two things are worth highlighting. First, ctrl.SetControllerReference sets an owner reference on the child resource, pointing back to the WebApp. Kubernetes garbage collection uses this: when the WebApp is deleted, the Deployment and Service are deleted automatically. You get cascade deletion for free.
Second, notice the pattern: fetch, check if not found, create if missing, patch if existing. This is the standard create-or-update idiom, and it is idempotent. Run it ten times with the same input and the cluster ends up in exactly the same state every time.
The reconcileService function follows the identical structure, substituting a Service object. It's not shown in full here to avoid repetition — the pattern is the same.
RBAC: what your controller needs and why
An Operator is a pod. Like any pod, it has a ServiceAccount. That ServiceAccount must be granted exactly the permissions the controller needs to do its job. Nothing more.
In Kubebuilder, RBAC is generated from the comment annotations on the reconciler function (the ones starting with // +kubebuilder:rbac: at the top of the reconciler). Run make manifests and Kubebuilder generates a ClusterRole and ClusterRoleBinding in config/rbac/.
The generated ClusterRole for our WebApp Operator looks like this:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: webapp-operator-role
rules:
- apiGroups: ["apps.codelooru.com"]
resources: ["webapps"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps.codelooru.com"]
resources: ["webapps/status"]
verbs: ["get", "update", "patch"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
resources: ["services"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]Each line reflects a concrete thing the reconciler does. It reads WebApp resources, so it needs get/list/watch. It writes status back, so it needs update/patch on the status subresource. It creates Deployments and Services, so it needs the full write verbs on those types. An Operator that only reads certain resource types should only have get/list/watch for those types.
The ClusterRoleBinding binds this role to the ServiceAccount that the Operator's pod runs as:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: webapp-operator-rolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: webapp-operator-role
subjects:
- kind: ServiceAccount
name: webapp-operator-controller-manager
namespace: webapp-operator-systemOne common mistake is granting ClusterRole access when the Operator only operates in one namespace. If your Operator is namespace-scoped, use a Role and RoleBinding instead. The difference matters for security audits in multi-tenant clusters.
Running and deploying
Running locally against a live cluster
The fastest feedback loop during development is running the controller outside the cluster, pointed at a live cluster via your local kubeconfig. This avoids rebuilding a container image on every change.
# Install the CRD into the cluster
make install
# Run the controller locally (uses ~/.kube/config)
make runThe controller connects to the API server, starts watching for WebApp resources, and processes events. You can now create a test resource:
kubectl apply -f - <<EOF
apiVersion: apps.codelooru.com/v1alpha1
kind: WebApp
metadata:
name: test-app
namespace: default
spec:
image: nginx:1.25
replicas: 2
port: 80
EOFWatch the controller logs. You should see the reconcile function fire, and within seconds a Deployment and Service named test-app should appear in the default namespace.
Deploying to the cluster
For production (or any shared environment), the controller runs as a pod. The flow is: build a container image, push it, then deploy the manifests Kubebuilder generated.
# Build and push the image
make docker-build docker-push IMG=your-registry/webapp-operator:v0.1.0
# Deploy the CRD, RBAC, and the Operator Deployment
make deploy IMG=your-registry/webapp-operator:v0.1.0The make deploy target applies everything under config/: the CRD, the ClusterRole, the ClusterRoleBinding, the ServiceAccount, and a Deployment for the controller pod itself. The controller pod runs in its own namespace (webapp-operator-system by default) and is the only thing that needs to exist for the Operator to work cluster-wide.
Checking your work
Once the Operator is running and you've created a WebApp, verify it with:
# The custom printcolumns from the +kubebuilder:printcolumn annotations show here
kubectl get webapp
# NAME IMAGE REPLICAS READY
# test-app nginx:1.25 2 2
# Verify child resources were created
kubectl get deployment test-app
kubectl get service test-app
# Check owner references — the Deployment should list test-app WebApp as owner
kubectl get deployment test-app -o jsonpath='{.metadata.ownerReferences}'
# Verify cascade deletion
kubectl delete webapp test-app
kubectl get deployment test-app # should be goneThe cascade deletion test is important. If owner references are set correctly, deleting the WebApp takes the Deployment and Service with it. If the child resources outlive the parent, owner references are missing and your cleanup logic needs attention.
To test the reconciler's correction behaviour, manually edit the Deployment to change the replica count, then watch the Operator correct it back. This is the core property you are testing: the controller should always win against manual drift.
kubectl scale deployment test-app --replicas=0
# Wait a few seconds — the Operator should restore it to spec.replicasSummary
An Operator is a two-file problem at its core. One file defines a struct (or POJO) that becomes a CRD. The other file implements a reconciler that fetches the custom resource, builds the desired state of child resources, and creates or patches them to close the gap. Everything else — the scaffolding, the RBAC, the leader election, the retry queue — is generated or handled by the framework.
The discipline is in the reconciler. It must be idempotent. It must handle not-found as a normal case, not an error. It must set owner references so cascade deletion works. It should write status back so users can observe what the Operator is doing. The framework handles the event queue, retry logic, and leader election; the reconciler contains only your domain logic.
The example here is simple by design. The same skeleton scales to arbitrarily complex systems. A database Operator might check replication lag before allowing an upgrade, or call a backup API before deleting a pod. A certificate Operator might watch expiry timestamps and trigger renewals proactively. The reconciliation loop is always the same shape — the intelligence lives entirely in what you put inside it.
Related: Explained: Kubernetes Operators — the conceptual foundation this post builds on. Part of the Explained series — concepts in tech, clearly.