Running Coder in a K3s Cluster Self-Hosted

on May 23rd, 2023

This post aims at describing the deployment of Coder on a Kubernetes cluster via the K3s distribution.

Coder is an online development environment platform that offers the possibility to code directly from a web browser. All the compilation toolchain is then deported on the server. Coder provides a ready-to-use remote development environment and reduces the installation and configuration issues that could arise from the heterogeneity of developer's workstation architectures and software used.

This kind of solution has gained popularity in recent years, with several alternative solutions such as:

These alternative solutions have a lot in common, but also some distinguishing features that can make them attractive.

I am interested in deploying an online development platform within a higher education governmental institution. This platform is meant to be used by researchers and students. Several features are mandatory, such as self-hosting, open source, great community, ease of installation, and the ability to handle multiple programming languages.

A first experimentation was based on Gitpod. Unfortunately, Gitpod no longer supports self-hosting, as explained on the Gitpod blog. Furthermore, setting up Gitpod was not easy, and I can understand the choice made by Gitpod developers, as explained on the blog post, to drop support for self-hosting since there were many issues arising from the heterogeneity of Kubernetes distributions. A second solution was JupyterHub, which was fairly easy to install but limited to the Python language. In a third experimentation, I therefore turned to Coder. In this post I will explain how I deployed it on a K3s cluster.

Coder solution matches all features I need, because it is self-hosted, open source and has a greatDiscord community. From a technical point of view, Coder provisions remote development environments via Terraform to supply users with Workspaces. For example, via Terraform, you can specify that you want to use a specific Docker image that contains the necessary compilation toolchain to build Java programs and that you want to install VIM as a code editor. Resources will also be specified, such as the memory or storage capacity of Workspaces. As for K3s, it is a lightweight Kubernetes distribution designed for production environments with limited resources, such as embedded systems. I appreciate K3s because it is designed to be easy to install.

The deployment of Coder on a K3s cluster is broken down into the following steps:

Setup
Install k3s
Deploy Coder
Deploy reverse-proxy
- NGINX
- Apache HTTP
Run and enjoy
Next steps

All materials can be found on my Github repository: https://github.com/mickaelbaron/coder-k3s-guide

Setup

Before starting the Coder install process, you must have:

The following infrastructure (to reproduce the experimentation):
- Three Ubuntu 22.04 machines with SSH credentials.
- The hostname of each machine (physical or virtual) is: k3sserver, k3snode1 and k3snode2.
- All machines are on the same vlan.
- All machines have ports 22 (SSH), 80 (HTTP), 443 (HTTPS) and 6443 (Kubernetes) exposed.
- A domain (coder.mydomain.com), a subdomain (*.coder.mydomain.com) and a configured DNS to redirect to k3sserver.
- A reverse-proxy (Apache HTTP or NGINX) which will be hosted outside of the Kubernetes cluster.
- A Docker installation on k3sserver to deploy the reverse-proxy.
Locally
- kubectl
- HELM

Note: in the following infrastructure, I will present a configuration where TLS certificates will be managed by a reverse-proxy outside of Kubernetes. It is a restriction of my higher education governmental institution and I would like to reproduce the same setup and to show you this specific use case. But, I agree in many cases, the ingress could/should also be secured via cert-manager or by passing TLS certificates in directly. It would be better to stay within the K8s infrastructure.

Install K3s

Connect to the server node (k3sserver) and run:

$ curl -sfL https://get.k3s.io | sh -

Extract the K3s token on the server node:

$ sudo cat /var/lib/rancher/k3s/server/node-token
K20545dbddda0f19bf1c9ac794546d200cdc4ede3fe9ad82d5e560ad0748cc28fd4::server:17a174d18d4fd82c0f99b687bd9aabcd

This is my own K3s token. You will have to adapt the next few instructions with YOUR K3s token.

Connect to the first agent node (k3snode1) and run:

$ export K3S_TOKEN=K20545dbddda0f19bf1c9ac794546d200cdc4ede3fe9ad82d5e560ad0748cc28fd4::server:17a174d18d4fd82c0f99b687bd9aabcd
$ curl -sfL https://get.k3s.io | K3S_URL=https://k3sserver:6443 sh -

Connect to the second agent node (k3snode2) and execute the same command line:

$ export K3S_TOKEN=K20545dbddda0f19bf1c9ac794546d200cdc4ede3fe9ad82d5e560ad0748cc28fd4::server:17a174d18d4fd82c0f99b687bd9aabcd
$ curl -sfL https://get.k3s.io | K3S_URL=https://k3sserver:6443 sh -

To get the cluster access file (k3s.yaml), from your host, run:

$ scp k3sserver:/etc/rancher/k3s/k3s.yaml .

Update the K3s server address (old value: 127.0.0.1) with the new hostname:

$ sed -i '' "s/127.0.0.1/k3sserver/" k3s.yaml

Check the K3s cluster:

$ export KUBECONFIG=$PWD/k3s.yaml
$ kubectl get nodes
NAME        STATUS   ROLES                  AGE   VERSION
k3sserver   Ready    control-plane,master   21d   v1.25.6+k3s1
k3snode1    Ready    <none>                 21d   v1.25.6+k3s1
k3snode2    Ready    <none>                 21d   v1.25.6+k3s1

These instructions are based on the K3s website: https://docs.k3s.io/quick-start.

Deploy Coder

Create a namespace for Coder, named coder in this example:

$ kubectl create namespace coder
namespace/coder created

Deploy PostgreSQL on the K3s cluster from the Bitnami repository:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories

$ helm install coder-db bitnami/postgresql \
    --namespace coder \
    --set auth.username=coder \
    --set auth.password=coder \
    --set auth.database=coder \
    --set persistence.size=10Gi

NAME: coder-db
LAST DEPLOYED: Wed Feb  31 09:53:58 2666
NAMESPACE: coder
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: postgresql
CHART VERSION: 11.1.28
APP VERSION: 14.2.0

** Please be patient while the chart is being deployed **

PostgreSQL can be accessed via port 5432 on the following DNS names from within your cluster:

    coder-db-postgresql.coder.svc.cluster.local - Read/Write connection

To get the password for "postgres" run:

    export POSTGRES_ADMIN_PASSWORD=$(kubectl get secret --namespace coder coder-db-postgresql -o jsonpath="{.data.postgres-password}" | base64 --decode)

To get the password for "coder" run:

    export POSTGRES_PASSWORD=$(kubectl get secret --namespace coder coder-db-postgresql -o jsonpath="{.data.password}" | base64 --decode)

To connect to your database run the following command:

    kubectl run coder-db-postgresql-client --rm --tty -i --restart='Never' --namespace coder --image docker.io/bitnami/postgresql:14.2.0-debian-10-r88 --env="PGPASSWORD=$POSTGRES_PASSWORD" \
      --command -- psql --host coder-db-postgresql -U coder -d coder -p 5432

    > NOTE: If you access the container using bash, make sure that you execute "/opt/bitnami/scripts/entrypoint.sh /bin/bash" in order to avoid the error "psql: local user with ID 1001} does not exist"

To connect to your database from outside the cluster execute the following commands:

    kubectl port-forward --namespace coder svc/coder-db-postgresql 5432:5432 &
    PGPASSWORD="$POSTGRES_PASSWORD" psql --host 127.0.0.1 -U coder -d coder -p 5432

Verify that a PostgreSQL pod has been created:

$ kubectl get pods --namespace coder
NAME                     READY   STATUS    RESTARTS   AGE
coder-db-postgresql-0    1/1     Running   0          22d

The cluster-internal DB URL for the PostgreSQL database is:

postgres://coder:[email protected]:5432/coder?sslmode=disable

Create a secret with the database URL:

$ kubectl create secret generic coder-db-url -n coder --from-literal=url="postgres://coder:[email protected]:5432/coder?sslmode=disable"

Add the Coder Helm repository:

$ helm repo add coder-v2 https://helm.coder.com/v2

Create a values.yaml configuration file with the suitable settings for your deployment. You should at least update the content following the # TODO comments:

coder:
  env:
    - name: CODER_PG_CONNECTION_URL
      valueFrom:
        secretKeyRef:
          name: coder-db-url
          key: url

    - name: CODER_ACCESS_URL
      # TODO
      value: "https://coder.mydomain.com"
    - name: CODER_WILDCARD_ACCESS_URL
      # TODO
      value: "*.coder.mydomain.com"

  service:
    enable: true
    type: ClusterIP
    sessionAffinity: ClientIP
    externalTrafficPolicy: Cluster
    loadBalancerIP: ""
    annotations: {}

  ingress:
    enable: true
    className: ""
    # TODO
    host: "coder.mydomain.com"
    # TODO
    wildcardHost: "*.coder.mydomain.com"
    annotations: {}
    tls:
      enable: false
      secretNames: ""
      wildcardSecretName: ""

The service will be configured as a ClusterIP. We configure Ingress to handle requests for the (coder.mydomain.com) domain.

Install the HELM chart on your K3s cluster:

$ helm install coder coder-v2/coder --namespace coder --values values.yaml
...

Check the pods into the coder namespace:

$ kubectl get pods --namespace coder
NAME                     READY   STATUS    RESTARTS   AGE
coder-db-postgresql-0    1/1     Running   0          22d
coder-59c6bc9c77-6f2wj   1/1     Running   0          9m47s

These instructions are based on the Coder website: https://coder.com/docs/v2/latest/install/kubernetes.

Deploy reverse-proxy

As mentioned in the setup section, a reverse-proxy will be deployed outside of your Kubernetes cluster.

The reverse-proxy will also be in charge of managing SSL/TLS certificates. Let's describe how to generate certificates with LetsEncrypt.

Connect to the server node (k3sserver) and install Certbot:

$ sudo apt-get update 
$ sudo apt-get install certbot -y

Create the SSL/TLS certificates:

$ sudo certbot certonly --agree-tos -m YOUR_EMAIL --manual --preferred-challenges=dns -d 'coder.mydomain.com' -d '*.coder.mydomain.com' -v
...

Copy the SSL/TLS certificates files (fullchain.pem and privkey.pem) into a directory (i.e. /ssl):

$ mkdir /ssl
$ cp /etc/letsencrypt/live/coder.mydomain.com-0001/fullchain.pem privkey.pem /ssl

Generate dhparams.pem file:

$ cd /ssl
$ openssl dhparam -out dhparams.pem 4096

You need to configure your Kubernetes cluster to update HTTP and HTTPS listen ports.

Connect to the server node (k3sserver) and create a /var/lib/rancher/k3s/server/manifests/traefik-config.yaml file with the following content:

kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ports:
      web:
        exposedPort: 8080
      websecure:
        exposedPort: 8443

Apply this configuration:

$ kubectl apply -f /var/lib/rancher/k3s/server/manifests/traefik-config.yaml

We suppose Docker is installed on the server node (k3sserver).

Create a Docker network called reverseproxynetwork:

$ docker network create reverseproxynetwork

Two reverse-proxy solutions will be presented: NGINX and Apache HTTP. Choose only ONE at your convenience.

NGINX

Connect to the server node (k3sserver).
Create an nginx directory:

$ mkdir ~/nginx

Create an NGINX configuration file nginx/conf/coder.conf with the following content:

server {
    listen       80;
    listen       [::]:80;
    server_name  *.coder.mydomain.com;
    return   301 https://$host$request_uri;
}

server {
   listen       443 ssl;
   server_name  *.coder.mydomain.com;

   ssl_protocols TLSv1.2 TLSv1.3;
   ssl_certificate /ssl/fullchain.pem;
   ssl_certificate_key /ssl/privkey.pem;
   ssl_dhparam /ssl/dhparam.pem;
   ssl_ecdh_curve secp384r1;
   ssl_prefer_server_ciphers on;
   ssl_ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH;

   location / {
       proxy_pass http://k3sserver:8080/;
       proxy_http_version 1.1;
       proxy_set_header Upgrade $http_upgrade;
       proxy_set_header Connection "Upgrade";
       proxy_set_header Host $host;
   }
}

Create a file nginx/docker-compose.yaml with the following content:

services:

  nginx:
    container_name: nginx
    image: nginx:latest
    volumes:
      - ./conf:/etc/nginx/conf.d
      - /ssl:/ssl
    restart: always
    ports:
      - "80:80"
      - "443:443"
    networks:
      - reverseproxynetwork

networks:
  reverseproxynetwork:
    name: reverseproxynetwork
    external: true

Create and start the NGINX container:

$ cd ~/nginx
$ docker compose up -d

Apache HTTP

Connect to the server node (k3sserver).
Create an apachehttp directory.

$ mkdir ~/apachehttp

Create an Apache HTTP configuration file apachehttp/conf/coder.conf with the following content:

<VirtualHost *:443>
    SSLEngine On
    SSLProxyEngine on

    SSLProxyVerify none
    SSLProxyCheckPeerCN off
    SSLProxyCheckPeerName off
    SSLProxyCheckPeerExpire off

    SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
    SSLHonorCipherOrder On
    SSLCipherSuite "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4 !SHA1 !SHA256 !SHA384"
    SSLCompression off

    # HSTS (http://fr.wikipedia.org/wiki/HTTP_Strict_Transport_Security)
    Header unset Strict-Transport-Security
    Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
    RequestHeader set X-Forwarded-Proto "https"
    RequestHeader set X-Forwarded-Port "443"

    # Certificates
    SSLCertificateFile /ssl/fullchain.pem
    SSLCertificateKeyFile /ssl/privkey.pem

    ServerName *.coder.mydomain.com

    ProxyPreserveHost On
    ProxyRequests off
    ProxyPass / http://k3sserver:8080/ upgrade=any
    ProxyPassReverse / http://k3sserver:8080/

    RewriteEngine on

    RewriteCond %{HTTP:Connection} Upgrade [NC]
    RewriteCond %{HTTP:Upgrade} websocket [NC]
    RewriteRule /(.*) ws://k3sserver:8080/$1 [P,L]

    # Custom log file for SSL
    ErrorLog /var/log/apachehttp/coder/error.log
    CustomLog /var/log/apachehttp/coder/access.log combined
</VirtualHost>

<VirtualHost *:80>
    ServerName *.coder.mydomain.com
    RewriteEngine On
    RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
</VirtualHost>

Copy and update at your convenience the apachehttp/httpd.conf configuration file.
Create a file apachehttp/docker-compose.yaml with the following content:

services:

  httpd:
    container_name: httpd
    image: httpd:latest
    volumes:
      - ./httpd.conf:/usr/local/apache2/conf/httpd.conf
      - ./conf:/usr/local/apache2/conf/sites
      - /ssl:/ssl
    restart: always
    ports:
      - "80:80"
      - "443:443"
    networks:
      - reverseproxynetwork

networks:
  reverseproxynetwork:
    name: reverseproxynetwork
    external: true

Create and start Apache HTTP container:

$ cd ~/apachehttp
$ docker compose up -d

Test Coder

Open the https://coder.mydomain.com URL with your favorite web browser.

Next steps

I hope this post helped you deploy Coder on a Kubernetes cluster via the K3s distribution.

I still have a lot to learn on how to use and configure Coder. In a future post, I will try to provide some feedback on my use of Coder and Kubernetes. Several questions still remain unanswered:

How should I scale? How much resources (cpu, memory, disk) should I allocate for each user (student or researcher)?
What are the limitations of using a remote development environment? How can I integrate hardware used during hands-on exercises (sensors, robots with Raspberry Pis)?
How can I provide access to users data, especially when hosted on remote volumes?

The answers to these questions, among others, might help me conclude whether Coder is the solution to my problem.

Subscribe to our Newsletter

Want to stay up to date on all things Coder? Subscribe to our monthly newsletter and be the first to know when we release new things!

Back To Blog

Discover how Dropbox streamlined dev environments & cut costs by switching 1000 developers to Coder