Jonathan Yu

Senior Software Engineer

Ben Potter

VP of Product

<p><strong>The challenge</strong></p>

<p>Continuous integration helps deliver complex software faster and more reliably. When failures occur, however, developers can try to reproduce the failure locally, or resort to diagnosing and resolving the problem by pushing additional changes to the pull request and waiting for a new build to complete, which results in long cycle times. In addition, they do not have access to familiar debugging tools, and will instead need to rely on non-interactive build logs:</p>


<p>The other option is for developers to reproduce the failure locally. To do so, however,  they often need to interpret and translate the configuration files used for the build, which can be complex and specific to a particular continuous integration system. Moreover, warnings or errors can be difficult to replicate due to subtle differences, such as versions of a tool or installed plugins.</p>

<p>Coder presents a third option: since it runs standard container images, you can use common images between development workspaces and build environments, eliminating <a href="https://coder.com/blog/what-is-configuration-drift">configuration drift</a>, not just between developers, but also between developers and the build system.</p>


<p><strong>Solution</strong></p>

<p>By moving developer workspaces to Coder, everyone on the team develops with a consistent set of tools, down to the exact version. Expanding upon this, we can reuse the same base image in build environments as well, reducing the divergence between dev and build infrastructure. When failures happen in the build system, developers can reproduce them in their personal development environments, confident in the knowledge that the tools and versions match.</p>

<p>For years, our team at Coder has been using this development/build/test workflow, greatly simplifying the process of debugging inevitable failures. When we have a lint check failure in our build environment, we can re-run the command in our development environment, comfortable in the knowledge that the output will be the same, since the tools are identical. For other checks that run frequently in continuous integration and rarely on local machines, such as browser-based tests, we ensure that local checks continue to succeed. In other words, our build/test infrastructure ensures that our development container image includes the exact tools necessary to build the software, acting in a virtuous cycle and ensuring dependencies are kept in sync.</p>


<p>Implementing this workflow with a system like GitHub Actions is simple: build a container image including the necessary development/build dependencies, push it to a container registry, and run builds using that image.</p>

<p><strong>Other benefits of this approach</strong></p>

<p>In addition to the primary benefit of keeping everything synchronized between development, build, and test environments, this workflow also:</p>

<ul>
<li>Improves portability of your build environment, since many build systems (including <a href="https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idcontainer">GitHub Actions</a>, <a href="https://docs.gitlab.com/ee/ci/docker/using_docker_images.html">GitLab CI</a>, <a href="https://buildkite.com/docs/tutorials/docker-containerized-builds">Buildkite</a>, and <a href="https://circleci.com/docs/2.0/custom-images/">CircleCI</a>) support running builds inside containers using an image you control</li>
<li>Allows the use of container security and compliance scanning tools (e.g., those produced by <a href="https://anchore.com/container-vulnerability-scanning/">Anchore</a>, <a href="https://jfrog.com/xray/">JFrog</a>, <a href="https://www.aquasec.com/">Aqua</a>, and <a href="https://snyk.io/product/container-vulnerability-management/">Snyk</a>) to help ensure that the container images used for development do not include vulnerable dependencies</li>
<li>Consolidates all dependencies for development and build into a Dockerfile, rather than spreading the instructions across documentation and setup scripts</li>
<li>Creates a fully-verifiable software supply chain through container signing with tools like <a href="https://www.sigstore.dev/">Cosign</a></li>
<li>Improves reliability and performance of the build process, since dependencies are “cached” in a container registry, rather than downloaded from various sources</li>
<li>Optionally enables “offline builds,” in which the container running the build is isolated from the Internet, further improving reproducibility and reducing risk of data exfiltration</li>
</ul>

<p>If you’d like to try Coder, you can <a href="https://coder.com/docs/coder/latest/setup/docker">self-host in minutes</a> with Docker. You can also <a href="https://coder.com/demo">request a demo</a> of Coder.</p>


Containerize your dev and build workflows to share dependencies and reduce configuration drift across environments.

Reduce CI failures with unified dev and build environments

Cian Johnston

Software Engineer

<p>Recently, <a href="https://www.openwall.com/lists/oss-security/2022/01/18/7">a vulnerability (CVE-2022-0185) was discovered in the Linux kernel</a> (versions 5.1 and above). This vulnerability allows a user with <strong>local access</strong> inside a <strong>non-privileged user namespace</strong> to gain <strong>root access</strong> by exploiting an integer underflow to gain the CAP<em>SYS</em>ADMIN capability in a new user namespace, which is normally reserved for processes running as the <strong>root</strong> user. </p>

<p><strong>This vulnerability affects users of Coder.</strong> </p>

<p>In order to exploit this vulnerability, a user must be logged into a Coder workspace.</p>

<p>Coder relies upon third-party components such as Docker, Sysbox, and Podman. These in turn rely heavily upon non-privileged user namespaces for security and isolation. The Nestybox team, for example, has <a href="https://github.com/nestybox/sysbox/commit/3a73a4bcdf5dcf1bb2c6357f8ba6a5cdc2c88b11">confirmed</a> that this issue affects downstream users.</p>

<p>One accepted mitigation strategy is to disable unprivileged user namespaces:</p>

<pre><code>sysctl -w kernel.unprivileged_userns_clone=0
</code></pre>

<p>However, this mitigation strategy may interfere with core system functionality Coder needs to work. We therefore <strong>cannot recommend</strong> this. We instead recommend that you <strong>update the Linux kernel</strong> on all systems that run Coder workspaces as soon as possible. </p>

<p>We also recommend that if you have updated to Coder version 1.27 already, you may wish to enable <a href="https://coder.com/docs/coder/latest/admin/workspace-management/process-logging">workspace process logging</a>, which will enable you monitor any attempts to exploit this vulnerability.</p>

<p>The Ubuntu and Red Hat kernel maintainers have released security bulletins for this issue:</p>

<ul>
<li>Ubuntu: <a href="https://ubuntu.com/security/CVE-2022-0185">https://ubuntu.com/security/CVE-2022-0185</a></li>
<li>Red Hat: <a href="https://access.redhat.com/security/cve/CVE-2022-0185">https://access.redhat.com/security/cve/CVE-2022-0185</a> </li>
</ul>

<p>If you are using a different Linux distribution to run Coder, please check the security bulletins for your distribution. We recommend that you check these periodically for new information on potential vulnerabilities, and install security patches as soon as they are available.</p>


This vulnerability affects users of Coder. Coder relies upon third-party components such as Docker, Sysbox, and Podman.

Statement on the recent CVE-2022-0185 vulnerability 

<p>Late last week, Chen Zhaojun of the Alibaba Cloud Security Team discovered a severe remote code execution vulnerability (<a href="https://nvd.nist.gov/vuln/detail/CVE-2021-44228">CVE-2021-44228</a>). This vulnerability affects many Java applications using the Log4j library. Security researchers at <a href="https://www.lunasec.io/docs/blog/log4j-zero-day/">LunaSec (which dubbed the vulnerability Log4Shell)</a>, <a href="https://www.fastly.com/blog/digging-deeper-into-log4shell-0day-rce-exploit-found-in-log4j">Fastly</a>, and <a href="https://blog.cloudflare.com/how-cloudflare-security-responded-to-log4j2-vulnerability/">Cloudflare</a> quickly published detailed analyses of the attack and how to mitigate the vulnerability.</p>

<p>While the Coder product does not use Java or Log4j for any of its services, many of the integrated development environments (IDEs) used with Coder, such as JetBrains’ IntelliJ IDEA, GoLand, and PyCharm, do use Java and the Log4j library. Coder’s security model enforces authentication and only permits users to access instances running in their own workspaces. As a result, workspaces running in Coder are unlikely to be affected by this vulnerability. We are working with our upstream vendors to investigate and patch any vulnerable applications to further safeguard against any security risks.</p>

<p><strong>Update:</strong> JetBrains has <a href="https://blog.jetbrains.com/blog/2021/12/13/log4j-vulnerability-and-jetbrains-products-and-services/">released a statement</a> noting all IntelliJ platform based IDEs and Gateway are not affected.</p>

<p>As a general risk mitigation strategy, we recommend using appropriate ingress and egress firewall rules, as well as Kubernetes network policies, to prevent data exfiltration. Coder uses Kubernetes <a href="https://coder.com/docs/coder/latest/setup/requirements#network-policies">network policies</a> to prevent workspaces from receiving ingress traffic directly, instead requiring connections to pass through an authenticating proxy, which includes access to IDEs, DevURL services, and tunnelled connections. Coder’s built-in security controls on DevURLs also permit <a href="https://coder.com/docs/coder/latest/admin/devurls#setting-dev-url-access-permissions">administrators to enforce an installation-wide policy</a>, such as requiring users to be authenticated using the organization’s identity provider.</p>


Late last week, a vulnerability in the Log4j package was discovered. Read our statement on Coder's security model with regards to this remote code execution vulnerability.

Statement on the recent Log4j security vulnerability

Erik Bledsoe

Technical Writer

<p><strong>Development teams deserve better than VDI</strong></p>

<p>The COVID-19 pandemic resulted in a huge increase in the use of Virtual Desktop Infrastructure (VDI) and its closely related cousin Desktop-as-a-Service (DaaS). A 2020 study <a href="https://www.globenewswire.com/news-release/2020/08/17/2079060/0/en/Impact-of-COVID-19-on-the-Virtual-Desktop-Infrastructure-VDI-Industry-2020-2027-Demand-from-IT-Telecom-Sector-Amid-COVID-19-Anticipated-to-Drive-Growth.html">predicts a compound annual growth rate of 14.4% for VDI</a> between 2019 and 2027, and Gartner forecasts <a href="https://www.gartner.com/smarterwithgartner/why-covid-19-drove-an-increase-in-daas">DaaS usage to grow over 150%</a> between 2020 and 2023.</p>

<p><strong>Organizations turn to VDI for a variety of reasons including:</strong> </p>

<ul>
<li>enabling remote work and mobility</li>
<li>simplifying desktop management </li>
<li>running arbitrary desktop applications</li>
<li>providing enhanced security </li>
<li>improving scalability and performance</li>
<li>onboarding and off-boarding contractors and external employees</li>
</ul>

<p>But for developers, the typical Windows-based VDI is overkill and adds unnecessary overhead. </p>

<h2>Coder vs. VDI: A Detailed Comparison</h2>

<p>A dedicated developer workspace platform like Coder meets the needs of the organization that drove its VDI adoption while providing developers with a better experience than VDI. System administrators benefit as well since the lighter-weight developer workspaces in Coder require fewer resources and can be packed more densely on your infrastructure.</p>

<h3>Enabling remote work and mobility</h3>

<p>Much of the recent growth in the use of VDI results from the sudden shift to remote work as a result of the pandemic, though broader trends are playing a part as well. </p>

<p>Like VDI, with Coder developer workspaces are centralized and <a href="https://coder.com/docs/coder/latest/setup/architecture">run within the organization’s infrastructure</a>. Developers can access their workspaces from wherever they are—their home, a coffee shop, the office—and using any device, <a href="https://coder.com/blog/i-developed-on-an-ipad-for-two-weeks">even an iPad</a>.</p>

<h3>Simplifying desktop management</h3>

<p>VDI allows IT to maintain a minimal number of master images. Rather than spending time troubleshooting an individual user’s machine, IT staff simply create a new virtual desktop allowing the user to resume work immediately. </p>

<p>With Coder, developer workspaces are created from a centrally managed image repository. Developers can create or rebuild existing workspaces or launch a new one with a click of a button, no need to open a ticket. And unlike most VDI implementations, developers’ <a href="https://coder.com/docs/coder/latest/workspaces/personalization">customizations to their workspaces persist</a> across rebuilds and automatically load with new workspaces.</p>

<h3>Providing enhanced security</h3>

<p>VDI is a common solution for organizations that prioritize a high level of security or are in highly regulated industries. </p>

<p>Coder applies the same principles as VDI in this regard: limit access to the organization’s resources and keep valuable information and data off of user endpoints. As a result, Coder is an ideal component for <a href="https://coder.com/whitepapers/zero-trust-security-for-software-development-teams">implementing zero-trust security for software development teams</a>. It even <a href="https://coder.com/docs/coder/latest/setup/air-gapped">fully functions in air-gapped networks</a>. </p>

<h3>Improving scalability and performance</h3>

<p>Organizations that are growing rapidly or that employ large numbers of seasonal workers often use VDI to avoid the bottleneck of acquiring physical devices. </p>

<p>Because Coder is deployed on Kubernetes, it can easily scale out to accommodate more users or scale in when the demand diminishes.</p>

<h3>Onboarding and off-boarding contractors and external employees</h3>

<p>Organizations that utilize large numbers of contractors use VDI to quickly provision and de-provision desktop resources as needed. This allows outside consultants to bring in their own laptops, access centralized desktops, and conduct their jobs, and for IT staff to quickly and efficiently cut off access at the end of the contract.</p>

<p>Again, Coder applies the same principles specifically for developers. By centralizing the management and orchestration of developer workspaces, Coder significantly reduces their onboarding times (whether they are contract or full time). Instead of being given access to a virtual machine and then needing to configure their development environment with all the dependencies required for a project (just as they would with a physical endpoint), developers can immediately begin coding in a preconfigured workspace. </p>

<p>Nearly all of Coder’s customers cite <a href="https://coder.com/blog/coder-customer-spotlight-kazoo">reduced onboarding times as a key factor in deciding to go with Coder</a>. </p>

<h2>Coder = A better experience for developers</h2>

<p>Coder clearly satisfies the business requirements that often drive the adoption of VDI. But while VDI is a general solution for running any application, Coder is optimized specifically for writing code. As a result, Coder provides a significantly better experience for developers.  Unlike VDI where all windows are within a single window, Coder allows developers to move IDEs, terminals, and other windows across all of their monitors. </p>

<h3>Faster builds and tests</h3>

<p>Coder allows developers to use the power and scale of the cloud to offload the burden of slow builds and tests from their local machine. While this is certainly true as well when moving the developer environment to VDI, Coder offers much more flexibility.</p>

<p>Because Coder enables “self-service” developer workspace creation, a developer could, for instance, increase the CPU and memory in their Coder workspace before beginning an intensive build or test. </p>

<p>Even better, Coder can also be <a href="https://coder.com/docs/coder/latest/guides/admin/resources">configured to share processor and memory resources between workspaces</a>, allowing workspaces to use idle processor and memory capacity while reducing total resource utilization.</p>

<h3>Reduced configuration drift</h3>

<p>How many hours have been wasted debugging a failed deployment after someone said “but it worked on my machine!”? </p>

<p>By ensuring that all members of the team are working from the same pre-configured image, <a href="https://coder.com/blog/what-is-configuration-drift">Coder reduces configuration drift</a>. Moreover, since workspaces run from an immutable container image, updating the application runtime or other dependencies is as simple as rebuilding the workspace, ensuring that the entire team is using a matching development environment. There is also a rich ecosystem of container signing and scanning tools, which ensures that containers are secure and contain up-to-date versions of system libraries and other applications.</p>

<p>While this is certainly also possible with VDI, it would be more difficult. VDI is intended to meet the general computing needs of the organization as a whole. Users sometimes expect full persistence of their environment, which provides a familiar experience but introduces the same challenges as managing end-user devices, particularly around ensuring patching, updates, and resolving software incompatibilities. As a result, developers who are required to use VDI are often provided a virtual desktop and told to configure their own environments on it, which inevitably leads to configuration drift and more “works on my machine” type problems.</p>

<h3>Low latency</h3>

<p>If you asked developers who are VDI averse to explain their reluctance, the most common answer would probably be the latency associated with VDI, particularly on slow networks or when experiencing high packet loss. </p>

<p>VDI works by rendering a full virtual desktop environment and sending it to the client. It transmits relatively large packets of data back and forth across the network for each keystroke or movement of the mouse.</p>

<p>While VDI vendors have made significant improvements in optimizing compression of these frames, the requirement for VDI systems to render a full graphical desktop inevitably results in higher load on the server and a perceptibly slower experience for users, particularly those working on a slower connection, such as those in the home. Many office applications, such as reading email, do not have a high degree of interactivity. In these cases, higher round-trip latency does not significantly degrade the user experience, but that’s not true for developer workspaces.</p>

<p>Coder works differently. When using VS Code in the browser, everything you type is rendered locally on your computer, then sent to the server to synchronize, significantly reducing the data packets that must travel back and forth, and thereby significantly reducing latency. Since it is purpose-built for software development, Coder can optimize the experience of writing code, much more intelligently than generic VDI solutions.</p>

<p>Developers can also run many IDEs locally and connect to their remote workspace through SSH, allowing them to take advantage of both the responsiveness of their local environment while offloading compute-intensive tasks like builds and tests to the remote workspace. </p>

<p>The result is a remote workspace that feels local.</p>


<h2>Comparison Chart</h2>

<table><thead>
<tr>
<th></th>
<th style="text-align: center">Coder</th>
<th style="text-align: center">VDI</th>
</tr>
</thead><tbody>
<tr>
<td>Remote access to workspace</td>
<td style="text-align: center">✅</td>
<td style="text-align: center">✅</td>
</tr>
<tr>
<td>Centralized management of workspaces</td>
<td style="text-align: center">✅</td>
<td style="text-align: center">✅</td>
</tr>
<tr>
<td>Keeps data off of endpoints</td>
<td style="text-align: center">✅</td>
<td style="text-align: center">✅</td>
</tr>
<tr>
<td>Well suited for zero-trust networks</td>
<td style="text-align: center">✅</td>
<td style="text-align: center">✅</td>
</tr>
<tr>
<td>Deploy in air-gapped environments</td>
<td style="text-align: center">✅</td>
<td style="text-align: center">✅</td>
</tr>
<tr>
<td>Easily scalable</td>
<td style="text-align: center">✅</td>
<td style="text-align: center">✅</td>
</tr>
<tr>
<td>Install on your own infrastructure, including on-premises</td>
<td style="text-align: center">✅</td>
<td style="text-align: center">✅</td>
</tr>
<tr>
<td>Portability across clouds</td>
<td style="text-align: center">✅</td>
<td style="text-align: center">Limited, may require nested virtualization</td>
</tr>
<tr>
<td>Faster onboarding</td>
<td style="text-align: center">✅</td>
<td style="text-align: center">✅</td>
</tr>
<tr>
<td>Pre-configured development workspaces</td>
<td style="text-align: center">✅</td>
<td style="text-align: center">Not usually</td>
</tr>
<tr>
<td>Self-serve workspace creation</td>
<td style="text-align: center">✅</td>
<td style="text-align: center">Not usually</td>
</tr>
<tr>
<td>Resource sharing during bursty workloads</td>
<td style="text-align: center">✅</td>
<td style="text-align: center">Not usually</td>
</tr>
<tr>
<td>Reduce configuration drift</td>
<td style="text-align: center">✅</td>
<td style="text-align: center">Not usually</td>
</tr>
<tr>
<td>Lowest latency</td>
<td style="text-align: center">✅</td>
<td style="text-align: center"></td>
</tr>
<tr>
<td>Run arbitrary desktop applications</td>
<td style="text-align: center">Through VNC</td>
<td style="text-align: center">✅</td>
</tr>
<tr>
<td>Integrate with cloud-native technologies, such as service meshes, advanced observability tools, security tools</td>
<td style="text-align: center">✅</td>
<td style="text-align: center">Not usually</td>
</tr>
<tr>
<td>Use container scanning tools to ensure system dependencies are up-to-date</td>
<td style="text-align: center">✅</td>
<td style="text-align: center">No</td>
</tr>
<tr>
<td>Use familiar container development tools, such as Docker</td>
<td style="text-align: center">✅</td>
<td style="text-align: center">✅</td>
</tr>
<tr>
<td>Licensing model</td>
<td style="text-align: center">Per user (seat)</td>
<td style="text-align: center">Sometimes per user; often per-host-CPU</td>
</tr>
<tr>
<td>Windows license required</td>
<td style="text-align: center">No, workspaces run on Linux</td>
<td style="text-align: center">Yes, as an additional purchase</td>
</tr>
</tbody></table>


<h2>See for yourself</h2>

<p>To learn more about how Coder can provide a better experience than VDI for your development team, <a href="https://coder.com/demo">request a demo</a>.</p>

<h2>Learn more</h2>

<p>See how Coder compares to GitHub Codespaces, Gitpod, VDI, and other solutions for remote development.</p>


Development teams deserve better than VDI. Coder provides an alternative to VDI for teams that require low latency, reduced configuration drift, faster builds, and more.

Coder: A better VDI alternative for developers

<p>Last week we hit the road for KubeCon + CloudNativeCon North America in Los Angeles for our first hybrid in-person and virtual conference. As a remote and globally-distributed team, it was fitting to use a variety of approaches to interact with users, partners, and open source project maintainers. As organizations mature in their adoption of cloud native technologies, security, observability, and developer velocity continue to be top of mind for developers and decision makers. We had far too many fantastic conversations to summarize in a single post, but here were some of our highlights.</p>

<h2>Supply chain security</h2>

<p>Software supply chain attacks continue to rise in both number and severity, with <a href="https://www.sonatype.com/resources/state-of-the-software-supply-chain-2021">Sonatype’s State of the Supply Chain report</a> describing a mind-boggling 650% year-over-year increase in the number of incidents in 2021. The <a href="https://www.whitehouse.gov/briefing-room/presidential-actions/2021/05/12/executive-order-on-improving-the-nations-cybersecurity/">Executive Order on Improving the Nation’s Cybersecurity in May</a> demonstrates that security continues to be a key risk for organizations, and while container technologies make it more difficult for attackers to maintain a foothold, the greater number and variety of applications in cloud native systems means that sophisticated tooling will be crucial. Luckily, the Linux Foundation, along with founding members Red Hat and Google, announced the open source <a href="https://www.sigstore.dev/">sigstore project</a> to make it easy for users to ensure provenance and integrity of third-party dependencies.</p>

<p>In addition, organizations including Snyk, Aqua Security, and Anchore remain at the forefront of container security and offer a variety of tools for publishing a full Software Bill of Materials. We’re also excited about <a href="https://chainguard.dev/">Chainguard, the recently-launched company founded by several industry luminaries</a>, since they share our passion for supply chain security and <a href="https://coder.com/blog/zero-trust-whitepaper">zero-trust approaches to security</a>.</p>

<h2>Zero-touch observability tools</h2>

<p>For cloud native applications, observability tools are essential for operating them effectively, and yet, developer teams often cannot afford to devote significant resources to implementing instrumentation. Leading firms including <a href="https://sysdig.com/">Sysdig</a>, <a href="https://pixielabs.ai/">Pixie Labs</a> (acq. New Relic), <a href="https://www.honeycomb.io/">Honeycomb</a>, <a href="https://isovalent.com/">Isovalent</a>, and <a href="https://www.rookout.com/">Rookout</a> are using advanced techniques such as eBPF or transparent reverse proxies to provide deep instrumentation of applications with minimal modifications to source code, providing the benefits of observability while significantly reducing their cost. These tools and platforms run the gamut from performance, security, networking, and debugging application logic.</p>

<h2>Container-native development experience</h2>

<p>Just as security and observability tools help us build better software, modern tools enable us to develop better software, faster. With Coder, your developer workspace runs in containers and you already have access to fast hardware that feels local. Sophisticated build tools including <a href="https://kompose.io/">kompose</a>, <a href="htts://tilt.dev">Tilt</a>, and <a href="https://garden.io">Garden</a> simplify development as well as testing of complex microservices applications in the cloud. These tools are a great complement to your developer workspace, since your project source code and containers can live together in the same infrastructure. </p>

<p>Moreover, <a href="https://loft.sh/docs/self-service/virtual-clusters">Loft’s virtual cluster</a> project enables developers to each have their own “personal” cluster, while keeping costs under control, since workloads actually run on a shared cluster with isolation at the Kubernetes control plane layer. These projects enable you to build complex applications in your existing Kubernetes infrastructure, supporting automatic re-builds and hot-reloading of modified code. Better yet, Tilt and Garden also provide instrumentation and observability features, simplifying debugging for complex applications.</p>

<h2>Other highlights</h2>

<p>As with our very own code-server project, open source software continues to drive innovation in the cloud native ecosystem. We met with Cosmonic, creators of the open source wasmCloud project, which enables developers to build applications in any language running anywhere, from data centers to the edge, powered by WebAssembly. We also met our friends and business partners over at Nestybox, the creators of sysbox, which is the open source container runtime that powers Coder’s container-based virtual machines, for the first time in person, after a great many discussions over Zoom and Slack.</p>

<p>We also took a break from juggling containers to try our hands at juggling:</p>


KubeCon Fun with Coder Swag

<h2>New in Coder v1.24</h2>

<p>We also released a new version of Coder last week. Here’s what’s new:</p>

<h4>AWS ECR support</h4>

<p>Customers can now host their Coder images using AWS Elastic Container Registry (ECR). See our documentation on adding private ECR Registries.</p>

<h4>Updates to Changelog</h4>

<p>Now you can view the changelog for up to 3 releases prior within the Coder dashboard.</p>

<h4>Service banner customization</h4>

<p>Coder’s service banner now supports custom colors. Admins can customize colors using a hex-code color picker to indicate various levels of severity for in-dashboard messages.</p>

<hr>

<p>For a full list of changes, see the <a href="https://coder.com/docs/coder/latest/changelog/1.24.0">1.24 changelog</a>.</p>


Last week we hit the road for KubeCon + CloudNativeCon North America in Los Angeles for our first hybrid in-person and virtual conference.

Highlights from KubeCon and Coder v1.24

Kyle Carberry

Co-founder and CTO

<p>WebRTC (Web Real-Time Communication) enables delivery of audio and video conferencing applications using native web technologies, but a lesser-known feature is that it can tunnel arbitrary data. This is the story of how and why we migrated our networking from a traditional reverse proxy architecture over to WebRTC, and what we learned in the process.</p>

<h2>Where we started</h2>

<p>Coder orchestrates development environments on your existing Kubernetes infrastructure, whether self-hosted or in the cloud. As with many cloud-native applications, Coder relied on a reverse proxy, which we called the “envproxy.” The envproxy routed traffic from outside the cluster into running workspaces. Coder is made up of a <em>control plane</em>—the service we call “coderd” which provides a dashboard for users to start and stop their workspaces—and a <em>data plane</em>⸺the envproxy and workspace.</p>


<p>In this context, the <em>data plane</em> refers to components that are in the critical path of the developer’s workflow; any failure would cause a disruption to a user’s development process. The proxy architecture provided a central point to enforce access control and perform audit logging, but it had consequences that led to a poor user experience:</p>

<ul>
<li>If the envproxy process crashes or restarts due to an upgrade, clients experience a brief interruption and programs need to re-establish any in-flight connections.</li>
<li>The proxy imposes additional round-trip latency, since it must read incoming stream data, decrypt it, and then re-encrypt it, so that we can forward it to the target workspace. As with audio or video calls on an unreliable connection, any visible latency when editing source code interactively results in <a href="https://input-delay.glitch.me/">similar frustration</a>.</li>
<li>The proxy operates at layer 7, constraining the protocols that developers can use to those that the proxy understands: for example, developers cannot access PostgreSQL databases in their workspace using its native binary protocol. The alternative is to use SSH port forwarding to tunnel these connections.</li>
<li>The proxy is an additional service to maintain, and since it is shared with multiple users of the system, any issues could impact multiple users simultaneously.</li>
<li>Services running inside the workspace containers need to listen for traffic from the proxy. While we use Network Policies to secure these, cluster administrators must ensure that the cluster networking enforces these policies. For example, Amazon’s Elastic Kubernetes Service (EKS) does not enforce these policies by default, and we want to take a defense-in-depth approach to enforce security boundaries.</li>
<li>Multiple points of failure meant that things could fail without being clearly diagnosable, and sometimes the failures would not be apparent to users.</li>
</ul>

<h2>What we evaluated</h2>

<p>Many of our engineers use SSH forwarding to tunnel PostgreSQL traffic or pass through an SSH agent socket from their local workstations to their Coder workspace. While this approach generally works well, the envproxy introduces a potential failure mode and inhibits upgrades; all in-flight connections must complete to gracefully shut down. A peer-to-peer architecture would resolve these challenges, while also providing a secure-by-default installation and reducing effort for system operators.</p>

<p>WireGuard solves the same fundamental challenge that we face with developer workspaces: securely providing end-to-end connectivity, across untrusted and unreliable networks, supporting a mesh operation mode and network roaming. Unfortunately, browsers do not include native support for WireGuard, and implementing it would be non-trivial, since browsers do not allow transmission of arbitrary UDP or even TCP traffic.</p>

<p>Our friends at <a href="https://blog.discord.com/how-discord-handles-two-and-half-million-concurrent-voice-users-using-webrtc-ce01c3187429">Discord use the WebRTC protocol to stream media in real time</a>, and we realized that Coder has similar requirements in terms of latency, security, browser and device support, and compatibility with diverse networking configurations. If we could tunnel arbitrary protocols over WebRTC, then we can leverage the existing ecosystem. Since browsers provide built-in WebRTC APIs, we would be able to modify our <a href="https://github.com/cdr/code-server">open source code-server project</a> to use it as an underlying transport, providing an even faster editing experience. Better still, we can provide end-to-end encryption and minimize latency on local networks, even behind gateways using Network Address Translation (NAT), through WebRTC-compatible technologies including STUN, TURN relays, and DTLS.</p>

<h2>Why and how we use WebRTC</h2>

<p>With our new networking model, Coder uses WebRTC to broker connections between the user and services running inside their workspace. In order to maximize compatibility and provide a clean migration path, our initial implementation uses the open source <a href="https://github.com/pion/turn/">Pion TURN relay server</a>. The relay provides a rendezvous point accessible to both the user and the workspace they are trying to access, and which may be located either inside or outside of the Kubernetes cluster, as long as it is able to receive inbound connections from both:</p>


<p>An agent running inside the workspace establishes an outbound connection to the relay service, and the user connects to the same relay with a token to authorize the connection to the endpoint. Both the user and the workspace must be able to connect to the relay, but it is not a requirement for the workspace container to connect directly to the user’s workstation or for the user’s workstation to connect directly to the workspace container. This approach means that:</p>

<ul>
<li>The workspace agent process can establish an outbound connection, rather than creating a listening port to accept inbound connections.</li>
<li>The proxy operates at the transport layer (the encrypted TCP stream between the workspace and relay service) rather than the application layer (SSH, HTTP, HTTPS), so it is protocol-independent and capable of tunnelling arbitrary protocols between the user and their workspace.</li>
<li>Only the relay service and control plane need to be accessible by clients, so we minimize the points of entry that administrators need to secure.</li>
</ul>

<h2>Looking forward</h2>

<p>While we believe that our new approach to networking will yield significant improvements to the experience of installing and operating a Coder deployment, we’re just getting started. In future releases, we want to explore:</p>

<ul>
<li>Now in Coder 1.22 (Aug 2021): Full peer-to-peer communication using STUN to reduce bandwidth demands on the TURN relay server and eliminate it as a single point of failure</li>
<li><a href="https://github.com/cdr/code-server/issues/2605">WebRTC support for code-server</a>, so that connections go directly between the user’s browser and development workspace, using STUN and TURN for NAT traversal</li>
<li>An in-browser WebRTC proxy using a web worker to transparently tunnel arbitrary HTTP/HTTPS network traffic</li>
<li>Testing of high-availability configurations to ensure that the networking remains reliable despite a variety of adverse network conditions</li>
<li>Using <a href="https://www.twilio.com/stun-turn">Twilio’s Global Network Traversal Service</a>, which hosts a global network of STUN and TURN relays, instead of the built-in service</li>
</ul>


<p>As a fallback when direct connectivity is not possible due to network conditions, traffic will flow through the TURN relay as depicted on the left. In future releases, users will be able to connect directly to their workspaces, relying on the STUN service to broker the connection and traverse NAT gateways. As a result, the relay will not be part of the data plane and interruptions to the STUN/TURN relay process will not affect connectivity to the workspace.</p>


The story of how and why Coder migrated our networking from a traditional reverse proxy architecture over to WebRTC, and what we learned in the process.

AI-native Development

Secure Environments for 2,500+ Developers

Jonathan Yu

Explore the latest from Jonathan

Reduce CI failures with unified dev and build environments

Statement on the recent CVE-2022-0185 vulnerability

Statement on the recent Log4j security vulnerability

Coder: A better VDI alternative for developers

Highlights from KubeCon and Coder v1.24

Rearchitecting Coder’s networking with WebRTC

Related content

Using Claude Code with Coder Workspaces

Introducing Coder Agents

Self-Hosted Doesn’t Always Mean What It Implies