Mike Terhar

Customer Success Engineer

<p><em>This content was <a href="https://vmblog.com/archive/2021/10/08/power-up-developers-with-mutant-pods.aspx">originally published on vmblog.com</a> and appears here with permission.</em></p>

<p>Kubernetes is being used for more than just scheduling containers to host production application workloads. As a generic compute and memory abstraction with network isolation and persistent storage, it can do a lot of stuff. One emerging use case is to run developer workspace images (aka development environment images) as pods. It’s breaking a lot of the classic “best practices,” but companies like Coder and GitPod have already proven its value in making developers’ lives easier.</p>

<p>There are several benefits to hosting developer workloads on Kubernetes ranging from cost-saving resource sharing to intellectual property boundaries. As Kubernetes brings more functionality to the platform, the developer workloads continue to gain capabilities. In this post, we’ll take a look at some of the advanced capabilities Kubernetes provides for powering up development environments.</p>

<h1>The pieces that make it possible</h1>

<h2>Pods</h2>

<p>When using Kubernetes to create developer workspaces, each workspace is a pod. The pod specification contains all of the information the scheduler needs to know to make the container run with the proper networking, storage, resource limits, image, environment variables, and secrets. </p>


<p>During scheduling, the pod goes through some evaluations that ensure the request is authorized and fits the allowed patterns. For a deeper dive on this, <a href="https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/">see the article</a> where this diagram comes from. Our focus for this post is on the use of mutating webhooks: receiving inbound requests and making adjustments to them.</p>

<h2>Webhooks</h2>

<p>There are two types of webhooks in Kubernetes: </p>

<ul>
<li>Validation webhooks check to see if a request  is allowed to happen. They’re responsible for preventing actions.</li>
<li>Mutating webhooks are applications that take API requests as an input and output a slightly modified request. </li>
</ul>

<p>Mutating webhooks are both incredibly powerful and incredibly dangerous. They can enable actions that the requestor may not realize are options. The baseline level of complexity is pretty high, despite some good templates existing. Since the webhooks stand in the way of inbound API requests, a buggy one could cause a lot of damage to the cluster. </p>


<h2>Gatekeeper</h2>

<p>Gatekeeper is a handy webhook intermediary. It is a rock solid product from the Open Policy Agent folks that started off as a validation webhook abstraction. It takes a validation spec and interprets that for you rather than having to code it yourself, letting Kubernetes admins control the behavior of webhooks through yaml files.</p>

<p>As of <a href="https://github.com/open-policy-agent/gatekeeper/releases/tag/v3.4.0">Gatekeeper 3.4</a> there is an experimental <a href="https://github.com/open-policy-agent/gatekeeper/blob/master/deploy/experimental/gatekeeper-mutation.yaml">mutation.yaml</a> feature that can be installed which starts up the gatekeeper mutations controller and adds some custom resources to make assignments. </p>

<h1>Warning: take care with mutating pods</h1>

<p>Just to be clear, this is a deviation from a core tenet of how Kubernetes has traditionally worked. Typically any time a pod spec is created, what goes in comes out exactly as defined. One could create a pod spec, apply it, and then validate that every field in the Kubernetes resource matched that of the file. This is no longer true when using a mutating webhook. Be sure not to mutate pods controlled by an operator or other mechanism that will detect the changes as erroneous and try to reset the specs. </p>

<h1>Mutating developer workspace pods</h1>

<p>With that background on Kubernetes and Gatekeeper behind us, let’s get into what we are really trying to do: <strong>make a developer’s life easier.</strong></p>

<p>In Kubernetes developer platforms like Coder, the central workspace scheduler is asked to create a new workspace for a developer with some amount of storage, memory, CPU, etc via a workspace pod. Some include lower level items like labels, annotations, or node selectors. Since Coder allows multiple clusters to be used from a single instance, developers may choose a cluster close to themselves for lower latency or closer to a cloud or datacenter resource for better execution performance. </p>

<p>These variables are a subset of the Kubernetes pod spec for a reason: user experience! Developers want their IDE to run smoothly and their code to compile error-free. They don’t want to be in the business of selecting the right values for CPUs, memory, and worrying about costs or other variables. In an attempt to simplify the end-user experience, some flexibility is removed to reduce the cognitive load on developers. </p>

<p>For some of these values, it makes sense to configure them at the workspace template level or user-specific configurations. Others may apply to everyone on the whole system as a policy. These are evolving and each product in this space is finding their preferred mixture of flexibility and intuitive user experience. </p>

<h1>The power of mutation</h1>

<p>In the meantime, there are a chunk of very powerful options that don’t make sense to expose within the developer UI but are very helpful when certain patterns arise. </p>

<p>Below are a few examples of things a developer might want to do and how they could be addressed today.</p>


<table><thead>
<tr>
<th>Developer Need</th>
<th>Generic pod method</th>
<th>Mutated Pod Method</th>
</tr>
</thead><tbody>
<tr>
<td>Run CUDA or other GPU accelerated math</td>
<td>Add a single GPU to a workspace, specified by the developer</td>
<td>Mutation can identify authorized developers and add any number of GPUs</td>
</tr>
<tr>
<td>Perform embedded software tests on attached hardware</td>
<td>No way for a user to specify a hostPath mount or guarantee the right node</td>
<td>Mutation can add a nodeselector and admin-controlled hostpath for the device</td>
</tr>
<tr>
<td>Use a special runtime with additional capabilities like Sysbox</td>
<td>The default runtime would have to be changed and all workspaces in a provider would be modified</td>
<td>Mutation can identify pods that need sysbox and modify the runtime for just those workspaces</td>
</tr>
<tr>
<td>Access a large dataset in a network share</td>
<td>FTP or SCP the files into the workspace, delays and duplication</td>
<td>Mutation can mount an NFS share to allow reading from a shared dataset</td>
</tr>
</tbody></table>


<p>As you can see from the pattern here, the optionality may not be provided because of security, cost, or complexity. The need doesn’t go away just because the product has trouble surfacing the capability in a secure and intuitive way. </p>

<p>Webhooks can only be created and modified by a cluster administrator which means the developers aren’t given the ability to create arbitrary mounts or GPUs. </p>

<p>The scope for each of these examples is cluster bound, where something innate to the cluster’s hardware or network location almost provides a capability.</p>

<h1>Example: Mount a shared volume</h1>

<p>Let’s demonstrate how to use a mutating webhook to mount a shared volume so that a developer will be able to access a large dataset without having to copy it into their workspace.</p>

<p>See the example on GitLab: <a href="https://gitlab.com/mterhar/mutating-admission">https://gitlab.com/mterhar/mutating-admission</a></p>


<p>Assume the NFS server that contains the huge dataset exists in a datacenter or cloud VPC along with a kubernetes cluster. This cluster has a namespace configured as a workspace provider. </p>

<p>In the current state, the workspace pods can’t mount an NFS share and would need a cluster administrator to manually adjust the spec for any pods that need that mount. The dataset will need to be accessible via another file transfer protocol and copied to the workspace for processing. </p>

<p>First we need the NFS volume as a read, write, many volume:</p>

<pre><code class="bash">$ cat &lt;&lt;&lt;EOT | kubectl create -f -n coder-big-dataset -
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs-pv
spec:
  capacity:
    storage: 1000Gi
  accessModes:
    - ReadOnlyMany
  nfs:
    server: nfs-server.domain.tld
    path: &quot;/big-dataset&quot;
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nfs-pvc
spec:
  accessModes:
    - ReadOnlyMany
  storageClassName: &quot;&quot;
  resources:
    requests:
      storage: 1000Gi
EOT
</code></pre>

<p>This ReadOnlyMany volume can now be mounted to as many pods as we make and they can all get to the data but can’t modify it. </p>

<p>To configure the mutation webhook, we have to follow the regular gatekeeper installation instructions and add the experimental mutations.yaml as well. </p>

<p>Then we run a command to create the Gatekeeper Assign resources in the coder-big-dataset namespace.</p>

<pre><code class="bash">$ cat &lt;&lt;&lt;EOT | kubectl create -f -n coder-big-dataset -
apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: Assign
metadata:
  name: add-nfs-volume
spec:
  applyTo:
  - groups: [&quot;&quot;]
    kinds: [&quot;Pod&quot;]
    versions: [&quot;v1&quot;]
  match:
    scope: Namespaced
    kinds:
    - apiGroups: [&quot;*&quot;]
      kinds: [&quot;Pod&quot;]
    namespaces: [&quot;coder-big-dataset&quot;]
  location: &quot;spec.containers[name:*].volumeMounts[name:nfs-volume]&quot;
  parameters:
    assign:
      value:
        name: nfs-volume
        mountPath: &quot;/nfs&quot;
---
apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: Assign
metadata:
  name: add-nfs-mount
spec:
  applyTo:
  - groups: [&quot;&quot;]
    kinds: [&quot;Pod&quot;]
    versions: [&quot;v1&quot;]
  match:
    scope: Namespaced
    kinds:
    - apiGroups: [&quot;*&quot;]
      kinds: [&quot;Pod&quot;]
    namespaces: [&quot;coder-big-dataset&quot;]
  location: &quot;spec.volumes[name:nfs-volume]&quot;
  parameters:
    assign:
      value:
        name: nfs-volume
        persistentVolumeClaim:
          claimName: nfs-pvc
EOT
</code></pre>

<p>You’ll notice there are two mutations for this change. It’s because the location of the modifications is different. Each mutation runs before the pod spec is evaluated so they can individually be incomplete, as long as they ApplyTo/Match the same pods they’ll be aligned before validation. </p>

<p>Create a pod or a workspace to see the NFS share mounted to the <code>/nfs</code> path.</p>

<pre><code class="bash">~ coder(nfs-one) √                                                           
% df -h
Filesystem                                   Size  Used Avail Use% Mounted on
overlay                                       97G   14G   84G  15% /
tmpfs                                         64M     0   64M   0% /dev
tmpfs                                        7.9G     0  7.9G   0% /sys/fs/cgroup
nfs-server.domain.tld:/big-dataset   1000G   14G   986G  1% /nfs
/dev/sde                                      32G   62M   32G   1% /home/coder
tmpfs                                        7.9G     0  7.9G   0% /dev/shm
/dev/root                                     97G   14G   84G  15% /etc/hosts
tmpfs                                        7.9G     0  7.9G   0% /proc/acpi
tmpfs                                        7.9G     0  7.9G   0% /proc/scsi
tmpfs                                        7.9G     0  7.9G   0% /sys/firmware
~ coder(nfs-one) √                                                           
% exit
</code></pre>

<h2>Conclusion</h2>

<p>This explanation and example are illustrative but not exhaustive. The types of problems that need to be solved vary from one organization to the next. </p>

<p>Mutating webhooks are a fantastic way to help a developer workspace gain access to some hardware or network specific capability that they would otherwise miss out on. </p>

<p>For more information on developer workspaces, check out <a href="https://coder.com">https://coder.com</a> and the code-server project at <a href="https://github.com/cdr/code-server">https://github.com/cdr/code-server</a>. </p>


Using Gatekeeper, webhooks, and pod mutations to make developers’ lives easier.

AI-native Development

Secure Environments for 2,500+ Developers

Mike Terhar

Explore the latest from Mike

Power-up Developers with Mutant Pods

Related content

Using Claude Code with Coder Workspaces

Introducing Coder Agents

Self-Hosted Doesn’t Always Mean What It Implies