Reduce CI failures with unified dev and build environments

Jonathan Yu

  

Ben Potter

 on March 2nd, 2022

The challenge

Continuous integration helps deliver complex software faster and more reliably. When failures occur, however, developers can try to reproduce the failure locally, or resort to diagnosing and resolving the problem by pushing additional changes to the pull request and waiting for a new build to complete, which results in long cycle times. In addition, they do not have access to familiar debugging tools, and will instead need to rely on non-interactive build logs:

The other option is for developers to reproduce the failure locally. To do so, however, they often need to interpret and translate the configuration files used for the build, which can be complex and specific to a particular continuous integration system. Moreover, warnings or errors can be difficult to replicate due to subtle differences, such as versions of a tool or installed plugins.

Coder presents a third option: since it runs standard container images, you can use common images between development workspaces and build environments, eliminating configuration drift, not just between developers, but also between developers and the build system.

Solution

By moving developer workspaces to Coder, everyone on the team develops with a consistent set of tools, down to the exact version. Expanding upon this, we can reuse the same base image in build environments as well, reducing the divergence between dev and build infrastructure. When failures happen in the build system, developers can reproduce them in their personal development environments, confident in the knowledge that the tools and versions match.

For years, our team at Coder has been using this development/build/test workflow, greatly simplifying the process of debugging inevitable failures. When we have a lint check failure in our build environment, we can re-run the command in our development environment, comfortable in the knowledge that the output will be the same, since the tools are identical. For other checks that run frequently in continuous integration and rarely on local machines, such as browser-based tests, we ensure that local checks continue to succeed. In other words, our build/test infrastructure ensures that our development container image includes the exact tools necessary to build the software, acting in a virtuous cycle and ensuring dependencies are kept in sync.

Implementing this workflow with a system like GitHub Actions is simple: build a container image including the necessary development/build dependencies, push it to a container registry, and run builds using that image.

Other benefits of this approach

In addition to the primary benefit of keeping everything synchronized between development, build, and test environments, this workflow also:

• Improves portability of your build environment, since many build systems (including GitHub Actions, GitLab CI, Buildkite, and CircleCI) support running builds inside containers using an image you control
• Allows the use of container security and compliance scanning tools (e.g., those produced by Anchore, JFrog, Aqua, and Snyk) to help ensure that the container images used for development do not include vulnerable dependencies
• Consolidates all dependencies for development and build into a Dockerfile, rather than spreading the instructions across documentation and setup scripts
• Creates a fully-verifiable software supply chain through container signing with tools like Cosign
• Improves reliability and performance of the build process, since dependencies are “cached” in a container registry, rather than downloaded from various sources
• Optionally enables “offline builds,” in which the container running the build is isolated from the Internet, further improving reproducibility and reducing risk of data exfiltration

If you’d like to try Coder, you can self-host in minutes with Docker. You can also request a demo of Coder.