New
The Gartner Hype Cycle Emerging Technologies Report is out!

Download now

Home
/
Administration
/
External Provisioners

External Provisioners

External provisioners

By default, the Coder server runs built-in provisioner daemons, which execute terraform during workspace and template builds. However, there are sometimes benefits to running external provisioner daemons:

  • Secure build environments: Run build jobs in isolated containers, preventing malicious templates from gaining shell access to the Coder host.

  • Isolate APIs: Deploy provisioners in isolated environments (on-prem, AWS, Azure) instead of exposing APIs (Docker, Kubernetes, VMware) to the Coder server. See Provider Authentication for more details.

  • Isolate secrets: Keep Coder unaware of cloud secrets, manage/rotate secrets on provisioner servers.

  • Reduce server load: External provisioners reduce load and build queue times from the Coder server. See Scaling Coder for more details.

Each provisioner can run a single concurrent workspace build. For example, running 30 provisioner containers will allow 30 users to start workspaces at the same time.

Provisioners are started with the coder provisionerd start command.

Authentication

The provisioner daemon must authenticate with your Coder deployment.

Set a provisioner daemon pre-shared key (PSK) on the Coder server and start the provisioner with coder provisionerd start --psk <your-psk>. If you are installing with Helm, see the Helm example below.

Coder still supports authenticating the provisioner daemon with a token from a user with the Template Admin or Owner role. This method is deprecated in favor of the PSK, which only has permission to access provisioner daemon APIs. We recommend migrating to the PSK as soon as practical.

Types of provisioners

Provisioners have two important tags: scope and owner. Coder sets these tags automatically.

Organization-Scoped Provisioners

Organization-scoped Provisioners can pick up build jobs created by any user. These provisioners always have tags scope=organization owner="".

coder provisionerd start

User-scoped Provisioners

User-scoped Provisioners can only pick up build jobs created from user-tagged templates. User-scoped provisioners always have tags scope=owner owner=<uuid>. Unlike the other provisioner types, any Coder user can run user provisioners, but they have no impact unless there is at least one template with the scope=user provisioner tag.

coder provisionerd start \
  --tag scope=user

# In another terminal, create/push
# a template that requires user provisioners
coder templates push on-prem \
  --provisioner-tag scope=user

Provisioner Tags

You can use provisioner tags to control which provisioners can pick up build jobs from templates (and corresponding workspaces) with matching tags.

coder provisionerd start \
  --tag environment=on_prem \
  --tag data_center=chicago

# In another terminal, create/push
# a template that requires this provisioner
coder templates push on-prem \
  --provisioner-tag environment=on_prem

# Or, match the provisioner exactly
coder templates push on-prem-chicago \
  --provisioner-tag environment=on_prem \
  --provisioner-tag data_center=chicago

A provisioner can run a given build job if one of the below is true:

  1. The provisioner and job tags are both organization-scoped and both have no additional tags set,
  2. The set of tags of the build job is a subset of the set of tags of the provisioner.

This is illustrated in the below table:

Provisioner TagsJob TagsCan run job?
{"owner":"","scope":"organization"}{"owner":"","scope":"organization"}true
{"owner":"","scope":"organization"}{"environment":"on_prem","owner":"","scope":"organization"}false
{"environment":"on_prem","owner":"","scope":"organization"}{"owner":"","scope":"organization"}false
{"environment":"on_prem","owner":"","scope":"organization"}{"foo":"bar","owner":"","scope":"organization"}true
{"environment":"on_prem","owner":"","scope":"organization"}{"data_center":"chicago","foo":"bar","owner":"","scope":"organization"}false
{"data_center":"chicago","environment":"on_prem","owner":"","scope":"organization"}{"foo":"bar","owner":"","scope":"organization"}true
{"data_center":"chicago","environment":"on_prem","owner":"","scope":"organization"}{"data_center":"chicago","foo":"bar","owner":"","scope":"organization"}true
{"owner":"aaa","scope":"owner"}{"owner":"","scope":"organization"}false
{"owner":"aaa","scope":"owner"}{"owner":"aaa","scope":"owner"}true
{"owner":"aaa","scope":"owner"}{"owner":"bbb","scope":"owner"}false
{"owner":"","scope":"organization"}{"owner":"aaa","scope":"owner"}false

The external provisioner in the above example can run build jobs with tags:

  • environment=on_prem
  • data_center=chicago
  • environment=on_prem datacenter=chicago
  • environment=cloud datacenter=chicago
  • environment=on_prem datacenter=new_york

However, it will not pick up any build jobs that do not have either of the environment or datacenter tags set. It will also not pick up any build jobs from templates with the user tag set.

Example: Running an external provisioner with Helm

Coder provides a Helm chart for running external provisioner daemons, which you will use in concert with the Helm chart for deploying the Coder server.

  1. Create a long, random pre-shared key (PSK) and store it in a Kubernetes secret

    kubectl create secret generic coder-provisioner-psk --from-literal=psk=`head /dev/urandom | base64 | tr -dc A-Za-z0-9 | head -c 26`
    
  2. Modify your Coder values.yaml to include

    provisionerDaemon:
      pskSecretName: "coder-provisioner-psk"
    
  3. Redeploy Coder with the new values.yaml to roll out the PSK. You can omit --version <your version> to also upgrade Coder to the latest version.

    helm upgrade coder coder-v2/coder \
        --namespace coder \
        --version <your version> \
        --values values.yaml
    
  4. Create a provisioner-values.yaml file for the provisioner daemons Helm chart. For example

    coder:
      env:
        - name: CODER_URL
          value: "https://coder.example.com"
      replicaCount: 10
    provisionerDaemon:
      pskSecretName: "coder-provisioner-psk"
      tags:
        location: auh
        kind: k8s
    

    This example creates a deployment of 10 provisioner daemons (for 10 concurrent builds) with the listed tags. For generic provisioners, remove the tags.

    Refer to the values.yaml file for the coder-provisioner chart for information on what values can be specified.

  5. Install the provisioner daemon chart

    helm install coder-provisioner coder-v2/coder-provisioner \
        --namespace coder \
        --version <your version> \
        --values provisioner-values.yaml
    

    You can verify that your provisioner daemons have successfully connected to Coderd by looking for a debug log message that says provisionerd: successfully connected to coderd from each Pod.

Example: Running an external provisioner on a VM

curl -L https://coder.com/install.sh | sh
export CODER_URL=https://coder.example.com
export CODER_SESSION_TOKEN=your_token
coder provisionerd start

Example: Running an external provisioner via Docker

docker run --rm -it \
  -e CODER_URL=https://coder.example.com/ \
  -e CODER_SESSION_TOKEN=your_token \
  --entrypoint /opt/coder \
  ghcr.io/coder/coder:latest \
  provisionerd start

Disable built-in provisioners

As mentioned above, the Coder server will run built-in provisioners by default. This can be disabled with a server-wide flag or environment variable.

coder server --provisioner-daemons=0
See an opportunity to improve our docs? Make an edit.