Notifications

Notifications are sent by Coder in response to specific internal events, such as a workspace being deleted or a user being created.

Enable experiment

In order to activate the notifications feature on Coder v2.15.X, you'll need to enable the notifications experiment. Notifications are enabled by default starting in v2.16.0.

# Using the CLI flag
$ coder server --experiments=notifications

# Alternatively, using the `CODER_EXPERIMENTS` environment variable
$ CODER_EXPERIMENTS=notifications coder server

More information on experiments can be found here.

Event Types

Notifications are sent in response to internal events, to alert the affected user(s) of this event. Currently we support the following list of events:

Workspace Events

These notifications are sent to the workspace owner.

• Workspace Deleted
• Workspace Manual Build Failure
• Workspace Automatic Build Failure
• Workspace Automatically Updated
• Workspace Dormant
• Workspace Marked For Deletion

User Events

These notifications are sent to users with owner and user admin roles.

• User Account Created
• User Account Deleted
• User Account Suspended
• User Account Activated
• (coming soon) User Password Reset
• (coming soon) User Email Verification

These notifications are sent to the user themselves.

• User Account Suspended
• User Account Activated

Template Events

These notifications are sent to users with template admin roles.

• Template Deleted

Configuration

You can modify the notification delivery behavior using the following server flags.

Delivery Methods

Notifications can currently be delivered by either SMTP or webhook. Each message can only be delivered to one method, and this method is configured globally with CODER_NOTIFICATIONS_METHOD (default: smtp).

Premium customers can configure which method to use for each of the supported Events; see the Preferences section below for more details.

SMTP (Email)

Use the smtp method to deliver notifications by email to your users. Coder does not ship with an SMTP server, so you will need to configure Coder to use an existing one.

Server Settings:

Authentication Settings:

TLS Settings:

NOTE: you MUST use CODER_EMAIL_FORCE_TLS if your smarthost supports TLS on a port other than 465.

Send emails using G-Suite

After setting the required fields above:

1. Create an App Password using the account you wish to send from
2. Set the following configuration options:

   CODER_EMAIL_SMARTHOST=smtp.gmail.com:465
   CODER_EMAIL_AUTH_USERNAME=<user>@<domain>
   CODER_EMAIL_AUTH_PASSWORD="<app password created above>"
   

See this help article from Google for more options.

Send emails using Outlook.com

After setting the required fields above:

1. Setup an account on Microsoft 365 or outlook.com
2. Set the following configuration options:

   CODER_EMAIL_SMARTHOST=smtp-mail.outlook.com:587
   CODER_EMAIL_TLS_STARTTLS=true
   CODER_EMAIL_AUTH_USERNAME=<user>@<domain>
   CODER_EMAIL_AUTH_PASSWORD="<account password>"
   

See this help article from Microsoft for more options.

Webhook

The webhook delivery method sends an HTTP POST request to the defined endpoint. The purpose of webhook notifications is to enable integrations with other systems.

Settings:

Here is an example payload for Coder's webhook notification:

{
	"_version": "1.0",
	"msg_id": "88750cad-77d4-4663-8bc0-f46855f5019b",
	"payload": {
		"_version": "1.0",
		"notification_name": "Workspace Deleted",
		"user_id": "4ac34fcb-8155-44d5-8301-e3cd46e88b35",
		"user_email": "[email protected]",
		"user_name": "danny",
		"user_username": "danny",
		"actions": [
			{
				"label": "View workspaces",
				"url": "https://et23ntkhpueak.pit-1.try.coder.app/workspaces"
			},
			{
				"label": "View templates",
				"url": "https://et23ntkhpueak.pit-1.try.coder.app/templates"
			}
		],
		"labels": {
			"initiator": "danny",
			"name": "my-workspace",
			"reason": "initiated by user"
		}
	},
	"title": "Workspace \"my-workspace\" deleted",
	"body": "Hi danny\n\nYour workspace my-workspace was deleted.\nThe specified reason was \"initiated by user (danny)\"."
}

The top-level object has these keys:

• _version: describes the version of this schema; follows semantic versioning
• msg_id: the UUID of the notification (matches the ID in the notification_messages table)
• payload: contains the specific details of the notification; described below
• title: the title of the notification message (equivalent to a subject in SMTP delivery)
• body: the body of the notification message (equivalent to the message body in SMTP delivery)

The payload object has these keys:

• _version: describes the version of this inner schema; follows semantic versioning
• notification_name: name of the event which triggered the notification
• user_id: Coder internal user identifier of the target user (UUID)
• user_email: email address of the target user
• user_name: name of the target user
• user_username: username of the target user
• actions: a list of CTAs (Call-To-Action); these are mainly relevant for SMTP delivery in which they're shown as buttons
• labels: dynamic map of zero or more string key-value pairs; these vary from event to event

User Preferences

All users have the option to opt-out of any notifications. Go to Account -> Notifications to turn notifications on or off. The delivery method for each notification is indicated on the right hand side of this table.

Delivery Preferences

Enterprise

Premium

Administrators can configure which delivery methods are used for each different event type.

You can find this page under https://$CODER_ACCESS_URL/deployment/notifications?tab=events.

Stop sending notifications

Administrators may wish to stop all notifications across the deployment. We support a killswitch in the CLI for these cases.

To pause sending notifications, execute coder notifications pause.

To resume sending notifications, execute coder notifications resume.

Troubleshooting

If notifications are not being delivered, use the following methods to troubleshoot:

1. Ensure notifications are being added to the notification_messages table
2. Review any error messages in the status_reason column, should an error have occurred
3. Review the logs (search for the term notifications) for diagnostic information
   If you do not see any relevant logs, set CODER_VERBOSE=true or --verbose to output debug logs

Internals

The notification system is built to operate concurrently in a single- or multi-replica Coder deployment, and has a built-in retry mechanism. It uses the configured Postgres database to store notifications in a queue and facilitate concurrency.

All messages are stored in the notification_messages table.

Messages older than 7 days are deleted.

Message States

A notifier here refers to a Coder replica which is responsible for dispatching the notification. All running replicas act as notifiers to process pending messages.

• a message begins in pending state
• transitions to leased when a Coder replica acquires new messages from the database
  • new messages are checked for every CODER_NOTIFICATIONS_FETCH_INTERVAL (default: 15s)
• if a message is delivered successfully, it transitions to sent state
• if a message encounters a non-retryable error (e.g. misconfiguration), it transitions to permanent_failure
• if a message encounters a retryable error (e.g. temporary server outage), it transitions to temporary_failure
  • this message will be retried up to CODER_NOTIFICATIONS_MAX_SEND_ATTEMPTS (default: 5)
  • this message will transition back to pending state after CODER_NOTIFICATIONS_RETRY_INTERVAL (default: 5m) and be retried
  • after CODER_NOTIFICATIONS_MAX_SEND_ATTEMPTS is exceeded, it transitions to permanent_failure

See Troubleshooting above for more details.