Four Reference Architectures for CDEs: Kubernetes vs. VMs

Ben Potter

 on February 2nd, 2024

The term “cloud development environment” (CDE) is pretty ambiguous. If you are looking into setting up CDEs for your enterprise, it’s important to consider the different architectures. For example, you may be surprised to learn that the Slack engineering team (as well as 44% of Coder users) use virtual machines over containers as their primary development environment.

We’ll get into the benefits of virtual machines vs. Kubernetes a bit later in the post. There isn’t a one-size-fits all solution.

Where to run CDEs

When selecting (or building) a platform for cloud development environments, you should also consider where they are going to run. There are some fully-managed (SaaS) solutions that you can provision inside your cloud account, or hybrid approaches that even support on-prem environments.

SaaS (fully managed)

There are many SaaS solutions that are as simple as sign up, click button, get dev environment. If you are in a non-regulated industry and have relatively simple codebases (small-to-medium-sized projects), this is a great option. Replit, GitHub Codespaces, and GitPod are vendors in this space and make it quick to get in and coding. You get into an environment by visiting a URL.

Disclaimer: Coder is intentionally not a SaaS product, so our writing will likely bias towards suggesting running CDEs in your cloud/on-prem CDEs. We think it’s the best fit for large organizations.

In the cloud

If you use a cloud provider such as AWS, Google Cloud, Azure, or DigitalOcean, or are in the process of migrating to the cloud, running development environments in the cloud comes with many benefits. First, CDEs make great training or “best practices” environments with Docker, Kubernetes namespaces, ephemeral environments, cloud CLIs, and any other tools they need.

Second, if your staging and production environments run in the cloud, running CDEs in the same location lets developers access all the storage buckets, databases, and other resources necessary for development. Developers can get an exact replica of what they are deploying against without having to wait on pipelines, configure VPNs, or run mock data/servers locally that often drift from production.

The large cloud providers have solutions, such as Azure Devbox or Google Cloud Workstations. Or, you could always go with a cloud-agnostic vendor like Coder or DevPod.

On-premise datacenters

Your first impression might be “yuck,” but hear me out. Large enterprises have a significant amount of data and dependencies in on-prem datacenters, and plan to keep them there. This means a large number of developers rely on on-prem filesystems, networks, and databases to develop against. We’ve also found that older and more complex codebases benefit more from CDEs as they take a long time to clone, build, and have the most steps to set up manually.

Plus, cloud compute is expensive. Unlike other applications that are mission critical, cloud development environments are often designed to be ephemeral in nature, meaning the premiums we often see in the clouds for reliability and elasticity is less necessary.

Reference architectures

Now let’s get into the fun stuff!

Kubernetes deployments

If you are already familiar with and encouraging cloud-native development, running CDEs on Kubernetes is a natural fit. You also save on infrastructure cost compared to VMs as you can overprovision many workspaces on a single node. After all, most of the time, CDEs sit relatively idle except for when they need to burst to build.

In many ways, Kubernetes is a great target environment for cloud development environments. The typical architecture is a pod/deployment with a persistent volume claim mounted at the user’s home directory.

This balance of ephemerality and persistence allows developers to rely on the base image to get the latest version of their tools and languages (e.g. python3) while a persistent disk stores the developer’s unique source code and any IDE customizations.

Since Kubernetes wasn’t made with CDEs in mind, we’ve made several open source tools to make the experience more natural and secure. envbuilder is an image that can build and run Devcontainers in Kubernetes while envbox makes it possible to securely run Docker and system services within Kubernetes pods.

However, there are challenges running CDEs on Kubernetes. Even with proper resource management in place, users can encounter a “noisy neighbor” problem through limitations of the underlying nodes such as file watcher limits or network bandwidth. Plus, if you aren’t well-versed in Kubernetes, managing a CDE cluster’s dynamic disks, periodic spikes in usage, and active terminal sessions within pods probably isn’t the place to start!

Kubernetes namespaces

This is like the previous architecture, but even more cloud-native and 🪄magical

Instead of waiting on a pipeline, PR preview environments, or manual deployments, developers can get a remarkably fast inner development loop that matches production. Tools such as Garden, vCuster, Okteto, and Coder make it possible for developers to get their own namespace in Kubernetes to build and deploy against. This reduces a long cycle time, often up to 10 minutes, that developers have to typically wait on to test their changes.

While tools like Garden and Okteto aren’t exactly CDEs (they sync code from a local filesystem to a remote cluster), they give developers an extremely fast development loop and can be paired with CDEs like Coder for onboarding and IDE consistency.

See this template for provisioning Kubernetes namespaces as dev environments in Coder.

VMs with nested containers

This may sound strange, but this is actually the architecture behind GitHub Codespaces. Each workspace is a VM running a single nested container.

Source: Github Codespaces Overview

There are several benefits to this approach, primarily around isolation. There is no noisy neighbor, and each CDE gets its own dedicated CPU, memory, isolated Docker socket, and volume mounts. At the same time, the developer interacts primarily with the container, giving them a consistent, reliable, and ephemeral environment. VMs are expensive, but as CDEs, they can automatically shut down when not in use. A developer uses their environment about 6 hours each day for 5 days a week. Why pay for it 24/7?

We’ve published two Coder OSS templates, and are working on building a version of envbuilder targeted at VMs.

Traditional VMs

We found that a large number of developers in enterprise organizations are already using virtual machines for their developer environments, managed CDE platform or not. We worked with SlashData to run a survey of 233 enterprise engineers and 41% reported their primary development environment is a “virtual machine in any cloud,” with only 20% reporting their primary environment was local. The rest were through a CDE platform such as Coder or GitHub Codespaces).

If this is also the case for your organization, moving to a centralized CDE platform gives you standardized operating systems, reduced support costs, smaller security surface area, and reduced cloud cost.

However, attempting to migrate all of these VM workloads to containers may be a mistake. A significant number of workloads may not be designed to run in a container. For example, VMs often are a better fit for large monorepos, desktop applications, app development, or applications that require Linux kernel features.

Source: Remote Development at Slack. Each engineer can lease a VM in an autoscaling group.

You can still make images with dependencies pre-installed with tools such as Packer or EC2 Image Builder, but this isn’t necessarily required. If developers just need a fast, secure, virtual machine and are willing to manually run setup scripts, this can remove the burden of image management.

We have Coder OSS templates for AWS, Azure, GCP, and DigitalOcean. With Terraform, you can decide whether the entire virtual machine is persistent (starts/stops when not in use) or ephemeral (deletes when not in use) with a persistent external “home” volume mounted, similar to the Kubernetes model.

Bonus: Provision any infrastructure as a CDE

Cloud development environments are not one-size-fits all, and we’ve seen some pretty complex configurations that provision cloud secrets, S3 buckets, bare metal machines, NFS mounts, and much more. Here are some of our favorites from our OSS telemetry:

Coder is an open-source platform for managing cloud development environments at scale. By leveraging Terraform-based templates, Coder workspaces can be deployed as pods, virtual machines, containers on any cloud, or even on-prem. Check out the Coder OSS documentation.

Our enterprise product, built for global 2000 and government organizations, includes security and governance features such as audit log, RBAC, groups, and template policies. Request a demo if you’re interested in working with us!

Special thanks to Alan Barr for guidance and reviewing this post.