Cian Johnston

Software Engineer

<p>We’re thrilled to announce the first release of <a href="https://github.com/coder/envbuilder">Envbuilder</a>, an open-source project inspired by an idea from our co-founder, Kyle Carberry. Built on the Dev Container standard, Envbuilder allows developers to take control of their dependencies and environments without waiting on platform teams. If you&#39;re not familiar with Dev Containers, you can read a helpful summary <a href="https://containers.dev/overview">here</a>.</p>

<p>In this post, we’ll dig into what Envbuilder is, who it’s for, and how it works. If you’re looking for an example of how to use Envbuilder to run dev containers on Kubernetes, check out <a href="https://coder.com/blog/run-dev-containers-on-kubernetes-and-openshift-with-envbuilder">this blog post</a>.</p>

<h2>Why Envbuilder?</h2>

<p>What if you could just &quot;run&quot; a Dev Container in your Kubernetes cluster straight from a Git repository? Without Docker-in-Docker?</p>

<p>It&#39;s hard to keep your development experience consistent and smooth while scaling your developer platform. You might decide to build pre-approved, standardized images for your development teams, but you then face the prospect of maintaining all of those different images. Why not let your development teams do this themselves? Envbuilder allows users to define their own dependencies and environments, helping break through these barriers and scale past thousands of daily active users. And it doesn&#39;t need Docker-in-Docker to work!</p>

<p>You can integrate pre-approved tools from sources like <a href="https://jfrog.com/artifactory/">Artifactory</a> to streamline workflows, reduce bottlenecks, and accelerate project delivery—all while maintaining the highest levels of security! Envbuilder eliminates the delays and frustrations of traditional environment setups and empowers teams to customize their own workspaces and enhance their own development experience.</p>

<p>Envbuilder runs inside a small container image, but does not require a Docker daemon in order to build a dev container! Instead, it builds in-place and then gets out of your way. This is useful in environments where you may not have access to a Docker socket for security reasons, but still need to work with a container.</p>

<p>To keep builds quick, Envbuilder can also use a Docker registry as a cache to store intermediate build stages.</p>

<p>Finally, Envbuilder is open-source and free for anyone to use!</p>

<h2>Who should use Envbuilder?</h2>

<p>Envbuilder is a good fit for developer teams who are familiar with Docker and/or <a href="https://containers.dev/">dev containers</a> and already using containerized development environments. If you have a large number of projects with different toolchains, dependencies, or that depend on a particular Linux distribution, Envbuilder will make it easier to quickly context switch between projects.</p>

<p>Envbuilder is also a great fit for more restricted environments where you may not have access to a Docker daemon—it doesn&#39;t need one to work!</p>

<h2>How does Envbuilder work?</h2>


<p>Envbuilder will:</p>

<ol>
<li>Clone a Git repository that contains a devcontainer.json or Dockerfile</li>
<li>Build the Dev container/Dockerfile in-place</li>
<li>Execute a customizable command in the built environment once the build is complete</li>
</ol>

<p>If there is a problem building the Dev container, Envbuilder can instead use a &#39;fallback&#39; image. </p>

<p>If you&#39;re developing a new dev container, you can use Envbuilder in a local workflow simply by mounting in a folder containing your <code>devcontainer.json</code>. You can see an example <a href="https://github.com/coder/envbuilder/blob/main/docs/using-local-files.md">here</a> of how to do this.</p>

<p>Envbuilder can also leverage a Docker registry to store intermediate build stages, allowing them to be re-used later on if there are no changes. To further improve build speeds, we have also created a <a href="https://github.com/coder/terraform-provider-envbuilder">Terraform provider</a> that allows you to perform a &#39;dry-run&#39; build of a dev container and start your workspace directly from a pre-built image, if it exists. This enables CI workflows that ensure your development teams&#39; environments are always ready to go.</p>

<p>Envbuilder is distributed as a Docker image. The image is small (~74MB) and contains only a single binary. </p>

<p>Envbuilder leverages the following open-source utilities and libraries:
-  <a href="https://github.com/GoogleContainerTools/kaniko">Kaniko</a> allows building containers without the need for a Docker daemon.
- <a href="https://github.com/go-git/go-git">Go-Git</a> is a pure-Go library that allows Envbuilder to clone Git repositories without the need for a local <code>git</code> executable.</p>

<h2>Next Steps</h2>

<p>Try out <strong>Envbuilder</strong> for yourself by visiting our <a href="https://github.com/coder/envbuilder">GitHub repository</a>. Want to see how it fits into your development workflows? Let’s talk about how we can help you integrate Envbuilder into your processes and improve your developer experience.</p>


Built on the Dev Container standard, Envbuilder allows developers to take control of their dependencies and environments without waiting on platform teams.

Envbuilder is here: Enable developers to customize their environments with Dev Containers

<p>At Coder, we embarked on a major rewrite of our flagship product, culminating in a <a href="https://coder.com/blog/announcing-coder2">v2.0 release in late 2023</a>. Prior to this, we started an initiative to perform comprehensive load tests to proactively identify and fix issues that would block rollouts at large scale. Spoiler: it turns out that scaling is hard, so we kept doing it.</p>

<h2>What do we mean by &quot;scale test&quot;?</h2>

<p>If you look on the internet, the overall consensus is that a &quot;scale test&quot; is where you attempt to determine the effects of increasing user load on a given system, while a &quot;stress test&quot; is where you throw as much load you can at a system to see how much it can handle.
Our &quot;scale tests&quot; fall somewhere in between the two – given a Coder deployment with a certain amount of resources, we want to determine its ability to handle a given amount of load.</p>

<p>When we perform a scale test, we do the following:</p>

<ul>
<li>Spin up a real Kubernetes cluster,</li>
<li>Install a given version of Coder via Helm with a set amount of resources available,</li>
<li>Create a given number of users and workspaces in parallel,</li>
<li>Send traffic to and from the workspaces,</li>
<li>Clean up created resources,</li>
</ul>

<p>There&#39;s a bunch of other supporting work, but that&#39;s the gist of it. Scale testing is testing at scale.</p>


<h2>Why did we do this?</h2>

<p>You can unpack this question a number of ways:</p>

<h3>Why did we do this at all?</h3>

<p>We&#39;re a small company, and our internal dogfood deployment has at most 19 active users. Some deployments have thousands of active users! We&#39;re obviously not going to run into the same kinds of problems as these deployments, so it&#39;s important for us to validate that Coder can perform well at this scale.</p>

<h3>Why not just perform benchmarks?</h3>

<p>Benchmarks only test individual system components, and don&#39;t tell you what sort of behaviours you&#39;ll see at scale. Think of it as an analogue to unit tests versus integration tests – you don&#39;t just want one part of the system to perform well, you also want the whole system to perform well.</p>

<h3>Why Kubernetes?</h3>

<p>A number of reasons but the main ones are:</p>

<ol>
<li>Our largest customers deploy Coder on Kubernetes, so this allows us to directly validate that deployment architecture.</li>
<li>Kubernetes also makes it simple to scale our test deployment up and down as we test different amounts of load versus resources.</li>
</ol>

<p>We&#39;ll go into more detail about our Kubernetes scale testing environment in a later post.</p>


<h2>What sort of problems did we run into?</h2>

<p>Because scaling is hard, it also follows that testing at scale is hard:</p>

<ul>
<li>Tooling: We ended up needing to write <a href="https://github.com/coder/coder/tree/main/scaletest">our own tooling</a> to load test Coder. Just running ApacheBench against our JSON API wouldn&#39;t have cut it:

<ul>
<li>A single JSON payload won&#39;t trigger the complex series of behaviours required to stand up a running workspace,</li>
<li>Some aspects of our testing involve setting up a persistent Tailscale connection,</li>
<li>Finally, we have a <a href="https://pkg.go.dev/github.com/coder/coder/v2/codersdk">Go SDK</a>. Why not use it?</li>
</ul></li>
<li>Monitoring: As running a scale test produces tons of data (you are monitoring as you test, aren&#39;t you?), we quickly realized that the best choice was to use the industry-standard <a href="https://coder.com/docs/v2/latest/admin/prometheus">Prometheus</a> time series database to export any data we needed, and <a href="https://github.com/coder/coder/blob/main/scaletest/scaletest_dashboard.json">Grafana</a> to make sense of it all.</li>
<li>Cost: Even when we scale down resources to a bare minimum, performing scale tests is still relatively expensive. We very quickly invested in <a href="https://github.com/coder/coder/tree/main/scaletest/terraform">automation</a> to bring up and tear down the cluster as needed.

<ul>
<li>We also added the option to use preemptible nodes to save money while developing the automation. However, we would recommend against performing scale tests using preemptible nodes, as one or more nodes may be pre-empted without warning and cause the test to fail.</li>
</ul></li>
</ul>


<h2>Was it worth it all, in the end?</h2>

<p>Most definitely! We found many issues, both large and small, for example:</p>

<ul>
<li>Each running workspace agent periodically performs a number of HTTP requests back to Coder for various purposes such as reporting statistics. Some of the queries behind these endpoints suffered badly from database read amplification, and others didn&#39;t need to write to the database immediately at all. We created custom database queries to minimize the overhead of these frequently-used endpoints, and added batching where applicable to reduce the number of database queries. Finally, we also moved the communication between the agent and Coder to use <a href="https://storj.github.io/drpc/">dRPC</a>, which reduced the overhead of multiple HTTP requests.

<ul>
<li><a href="https://github.com/coder/coder/issues/8063">https://github.com/coder/coder/issues/8063</a></li>
<li><a href="https://github.com/coder/coder/issues/8064">https://github.com/coder/coder/issues/8064</a> </li>
<li><a href="https://github.com/coder/coder/issues/8065">https://github.com/coder/coder/issues/8065</a></li>
<li><a href="https://github.com/coder/coder/issues/9782">https://github.com/coder/coder/issues/9782</a></li>
<li><a href="https://github.com/coder/coder/issues/10278">https://github.com/coder/coder/issues/10278</a></li>
</ul></li>
<li>Workspace agents connect to Coder using Tailscale. Initially Coder was wired up to create a separate Tailscale network for each agent, which resulted in excessively high memory usage when dialing multiple agents from Coder at once. We modified our network overlay to use a single Tailscale network for all agents, which resulted in a massive decrease in memory usage in this scenario!

<ul>
<li><a href="https://github.com/coder/coder/issues/8071">https://github.com/coder/coder/issues/8071</a></li>
<li>Bonus: while testing this fix, we also found the fix needed a fix! <a href="https://github.com/coder/coder/issues/9929">https://github.com/coder/coder/issues/9929</a></li>
</ul></li>
<li>We quickly found that creating a large number of workspaces at once takes up a considerable amount of resources. Because of this, we modified our Helm chart to allow separating the workload of building workspaces from the main Coder deployment itself, which allows us (and you) to scale the provisioning of workspaces separately!

<ul>
<li><a href="https://github.com/coder/coder/issues/8243">https://github.com/coder/coder/issues/8243</a> </li>
</ul></li>
<li>While iterating on our <a href="https://github.com/coder/coder/tree/main/scaletest/templates/scaletest-runner">scaletesting template</a>, we found a bug in the template editor that would cause the executable bit of template files to be removed!

<ul>
<li><a href="https://github.com/coder/coder/issues/10034">https://github.com/coder/coder/issues/10034</a></li>
</ul></li>
</ul>


<h2>How do I run one myself?</h2>

<p>We have fairly comprehensive steps documented in our <a href="https://github.com/coder/coder/blob/main/scaletest/README.md">GitHub repository</a>, and we also have more detailed documentation about our <a href="https://coder.com/docs/v2/latest/admin/architectures#scale-testing-methodology">scale testing method</a>. But here&#39;s a quick version using KinD. Note that you will be constrained by the CPU and memory resources available on your host machine.</p>

<ul>
<li>Ensure you have KinD installed on your machine, if you have not done so already. You can download it from <a href="https://github.com/kubernetes-sigs/kind/releases">https://github.com/kubernetes-sigs/kind/releases</a>. You will need Docker installed and working on your host machine.</li>
<li>Create a local Kubernetes cluster: </li>
</ul>

<pre><code>        kind create cluster –name coder
</code></pre>

<ul>
<li>Ensure your kubectl is configured to speak to this cluster, this should report that the Kubernetes control plane is running at <a href="http://127.0.0.1:12345">http://127.0.0.1:12345</a> (Note: the port may differ for you):</li>
</ul>

<pre><code>        kubectl cluster-info
</code></pre>

<ul>
<li>Install Coder using the Helm chart, limiting Coder to 1 CPU core and 1 GB memory: </li>
</ul>

<pre><code>        helm repo add coder-v2 https://helm.coder.com/v2
</code></pre>

<pre><code>        helm repo update
</code></pre>

<pre><code>        helm install coder coder-v2/coder \ 
                --namespace coder \ 
                --create-namespace \ 
                --set coder.resources.limits.cpu=1 \ 
                --set coder.resources.limits.memory=1Gi 
</code></pre>

<ul>
<li>Initialize the Coder deployment and create the first user:</li>
</ul>

<pre><code>        kubectl --namespace coder exec deployment/coder -- \
                coder login \
                --first-user-username=admin \
                --first-user-email=admin@example.com \
                --first-user-password=SomeSecurePassw0rd \
                --first-user-trial=false
</code></pre>

<ul>
<li>At this point, you can now access the Coder dashboard by port-forwarding and opening <a href="http://localhost:8080:">http://localhost:8080:</a></li>
</ul>

<pre><code>        kubectl --namespace coder port-forward service/coder 8080:80 &amp;
</code></pre>

<ul>
<li>Import the default Kubernetes template. You can run this command or manually import it in the UI:</li>
</ul>

<pre><code>        kubectl --namespace coder exec deployment/coder -- \ 
                coder templates init \ 
                --id kubernetes /tmp/kubernetes
</code></pre>

<pre><code>        kubectl --namespace coder exec deployment/coder -- \ 
                coder templates push kubernetes \ 
                -d /tmp/kubernetes \ 
                --variable namespace=coder \ 
                --yes
</code></pre>

<ul>
<li>Create 3 scale test users and workspaces using the above template. The --no-cleanup flag prevents the workspaces from being deleted automatically on finish:</li>
</ul>

<pre><code>        kubectl --namespace coder exec deployment/coder -- \ 
                coder exp scaletest create-workspaces \ 
                --template kubernetes \ 
                --count=3 \ 
                --parameter cpu=2 \ 
                --parameter memory=2 \ 
                --parameter home_disk_size=1 \ 
                --no-cleanup
</code></pre>

<ul>
<li>Send SSH traffic to all workspaces in 128 byte chunks every 100ms for 60 seconds:</li>
</ul>

<pre><code>        kubectl --namespace coder exec deployment/coder -- \
                coder exp scaletest workspace-traffic \
                --concurrency=0 \
                --bytes-per-tick=128 \
                --tick-interval=100ms \
                --ssh \
                --timeout=60s
</code></pre>

<ul>
<li>Clean up all workspaces:</li>
</ul>

<pre><code>        kubectl --namespace coder exec deployment/coder -- \ 
                coder exp scaletest cleanup
</code></pre>

<ul>
<li>When you are finished, you can delete the KinD cluster:</li>
</ul>

<pre><code>        kind delete cluster --name coder
</code></pre>


<p>For more information on scale testing, you can see our <a href="https://coder.com/docs/v2/latest/admin/scale#scale-testing-utility">online documentation</a> or run coder exp scaletest –help. Note that the exp scaletest command is not included in the agent (aka. &quot;slim&quot; binary) to save space, so make sure you are running the full Coder binary by checking the output of coder --version.</p>


Explore how Coder's scale testing ensured a smooth v2.0 release. In "Scale Testing Part 1: Why Scaletest?" we detail our rigorous process of identifying and fixing scalability issues, using Kubernetes for testing, and the improvements made to ensure Coder handles large-scale deployments effectively.

Scale Testing Part 1: Why Scale test?

Scale Testing Part 1: Why Scale Test?

Jonathan Yu

Senior Software Engineer

<p>Recently, <a href="https://www.openwall.com/lists/oss-security/2022/01/18/7">a vulnerability (CVE-2022-0185) was discovered in the Linux kernel</a> (versions 5.1 and above). This vulnerability allows a user with <strong>local access</strong> inside a <strong>non-privileged user namespace</strong> to gain <strong>root access</strong> by exploiting an integer underflow to gain the CAP<em>SYS</em>ADMIN capability in a new user namespace, which is normally reserved for processes running as the <strong>root</strong> user. </p>

<p><strong>This vulnerability affects users of Coder.</strong> </p>

<p>In order to exploit this vulnerability, a user must be logged into a Coder workspace.</p>

<p>Coder relies upon third-party components such as Docker, Sysbox, and Podman. These in turn rely heavily upon non-privileged user namespaces for security and isolation. The Nestybox team, for example, has <a href="https://github.com/nestybox/sysbox/commit/3a73a4bcdf5dcf1bb2c6357f8ba6a5cdc2c88b11">confirmed</a> that this issue affects downstream users.</p>

<p>One accepted mitigation strategy is to disable unprivileged user namespaces:</p>

<pre><code>sysctl -w kernel.unprivileged_userns_clone=0
</code></pre>

<p>However, this mitigation strategy may interfere with core system functionality Coder needs to work. We therefore <strong>cannot recommend</strong> this. We instead recommend that you <strong>update the Linux kernel</strong> on all systems that run Coder workspaces as soon as possible. </p>

<p>We also recommend that if you have updated to Coder version 1.27 already, you may wish to enable <a href="https://coder.com/docs/coder/latest/admin/workspace-management/process-logging">workspace process logging</a>, which will enable you monitor any attempts to exploit this vulnerability.</p>

<p>The Ubuntu and Red Hat kernel maintainers have released security bulletins for this issue:</p>

<ul>
<li>Ubuntu: <a href="https://ubuntu.com/security/CVE-2022-0185">https://ubuntu.com/security/CVE-2022-0185</a></li>
<li>Red Hat: <a href="https://access.redhat.com/security/cve/CVE-2022-0185">https://access.redhat.com/security/cve/CVE-2022-0185</a> </li>
</ul>

<p>If you are using a different Linux distribution to run Coder, please check the security bulletins for your distribution. We recommend that you check these periodically for new information on potential vulnerabilities, and install security patches as soon as they are available.</p>


This vulnerability affects users of Coder. Coder relies upon third-party components such as Docker, Sysbox, and Podman.

AI-native Development

Secure Environments for 2,500+ Developers

Cian Johnston

Explore the latest from Cian

Envbuilder is here: Enable developers to customize their environments with Dev Containers

Scale Testing Part 1: Why Scale Test?

Statement on the recent CVE-2022-0185 vulnerability

Related content

Using Claude Code with Coder Workspaces

Introducing Coder Agents

Self-Hosted Doesn’t Always Mean What It Implies