# Prometheus Coder exposes many metrics which can be consumed by a Prometheus server, and give insight into the current state of a live Coder deployment. If you don't have a Prometheus server installed, you can follow the Prometheus [Getting started](https://prometheus.io/docs/prometheus/latest/getting_started/) guide. ## Enable Prometheus metrics [#enable-prometheus-metrics] Coder server exports metrics via the HTTP endpoint, which can be enabled using either the environment variable `CODER_PROMETHEUS_ENABLE` or the flag `--prometheus-enable`. The Prometheus endpoint address is `http://localhost:2112/` by default. You can use either the environment variable `CODER_PROMETHEUS_ADDRESS` or the flag `--prometheus-address :` to select a different listen address. If `coder server --prometheus-enable` is started locally, you can preview the metrics endpoint in your browser or with `curl`: ```console $ curl http://localhost:2112/ # HELP coderd_api_active_users_duration_hour The number of users that have been active within the last hour. # TYPE coderd_api_active_users_duration_hour gauge coderd_api_active_users_duration_hour 0 ... ``` ### Kubernetes deployment [#kubernetes-deployment] The Prometheus endpoint can be enabled in the [Helm chart's](https://github.com/coder/coder/blob/release/2.37/helm) `values.yml` by setting `CODER_PROMETHEUS_ENABLE=true`. Once enabled, the environment variable `CODER_PROMETHEUS_ADDRESS` will be set by default to `0.0.0.0:2112`. A Service Endpoint will not be exposed; if you need to expose the Prometheus port on a Service, (for example, to use a `ServiceMonitor`), create a separate headless service instead. ```yaml apiVersion: v1 kind: Service metadata: name: coder-prom namespace: coder spec: clusterIP: None ports: - name: prom-http port: 2112 protocol: TCP targetPort: 2112 selector: app.kubernetes.io/instance: coder app.kubernetes.io/name: coder type: ClusterIP ``` ### Prometheus configuration [#prometheus-configuration] To allow Prometheus to scrape the Coder metrics, you will need to create a `scrape_config` in your `prometheus.yml` file, or in the Prometheus Helm chart values. The following is an example `scrape_config`. ```yaml scrape_configs: - job_name: "coder" scheme: "http" static_configs: # replace with the the IP address of the Coder pod or server - targets: [":2112"] labels: apps: "coder" ``` If you run a [standalone AI gateway](/beta-docs/ai-coder/ai-gateway/standalone/), each replica exports its own metrics on its own listener. Its Helm chart uses the same `0.0.0.0:2112` default as the `coder` chart, but sets up no scrape discovery. Refer to [AI gateway monitoring](/beta-docs/ai-coder/ai-gateway/monitoring/#kubernetes-discovery) for more details. To use the Kubernetes Prometheus operator to scrape metrics, you will need to create a `ServiceMonitor` in your Coder deployment namespace. The following is an example `ServiceMonitor`. ```yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: coder-service-monitor namespace: coder spec: endpoints: - port: prom-http interval: 10s scrapeTimeout: 10s namespaceSelector: matchNames: - coder selector: matchLabels: app.kubernetes.io/name: coder ``` ## Available metrics [#available-metrics] You must first enable `coderd_agentstats_*` with the flag `--prometheus-collect-agent-stats`, or the environment variable `CODER_PROMETHEUS_COLLECT_AGENT_STATS` before they can be retrieved from the deployment. They will always be available from the agent. The `coder_ai_gateway_cost_control_*` metrics are exported only by `coderd`. | Name | Type | Description | Labels | | ------------------------------------------------------------------------ | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | `agent_boundary_log_proxy_batches_dropped_total` | counter | Total number of boundary log batches dropped before reaching coderd. Reason: buffer\_full = the agent's internal buffer is full, meaning boundary is producing logs faster than the agent can forward them to coderd; forward\_failed = the agent failed to send the batch to coderd, potentially because coderd is unreachable or the connection was interrupted. | `reason` | | `agent_boundary_log_proxy_batches_forwarded_total` | counter | Total number of boundary log batches successfully forwarded to coderd. Compare with batches\_dropped\_total to compute a drop rate. | | | `agent_boundary_log_proxy_logs_dropped_total` | counter | Total number of individual boundary log entries dropped before reaching coderd. Reason: buffer\_full = the agent's internal buffer is full; forward\_failed = the agent failed to send the batch to coderd; boundary\_channel\_full = boundary's internal send channel overflowed, meaning boundary is generating logs faster than it can batch and send them; boundary\_batch\_full = boundary's outgoing batch buffer overflowed after a failed flush, meaning boundary could not write to the agent's socket. | `reason` | | `agent_scripts_executed_total` | counter | Total number of scripts executed by the Coder agent. Includes cron scheduled scripts. | `agent_name` `success` `template_name` `username` `workspace_name` | | `coder_ai_gateway_circuit_breaker_rejects_total` | counter | Total number of requests rejected due to open circuit breaker. | `endpoint` `model` `provider` | | `coder_ai_gateway_circuit_breaker_state` | gauge | Current state of the circuit breaker (0=closed, 0.5=half-open, 1=open). | `endpoint` `model` `provider` | | `coder_ai_gateway_circuit_breaker_trips_total` | counter | Total number of times the circuit breaker transitioned to open state. | `endpoint` `model` `provider` | | `coder_ai_gateway_cost_control_blocked_requests_total` | counter | The number of AI requests blocked because the initiator's budget was exceeded. | `group_id` | | `coder_ai_gateway_cost_control_blocked_users` | gauge | The number of users currently over their AI budget. | `group_id` | | `coder_ai_gateway_cost_control_enforcement_duration_seconds` | histogram | The duration of AI budget enforcement checks, in seconds (outcome: allowed, blocked, error). | `outcome` | | `coder_ai_gateway_cost_control_unpriced_token_usage_records_total` | counter | The number of recorded AI token-usage records for which no (provider\_type, model) price was found. provider is the provider instance name, and provider\_type is its configured type. | `model` `provider_type` `provider` | | `coder_ai_gateway_injected_tool_invocations_total` | counter | The number of times an injected MCP tool was invoked by AI Gateway. | `model` `name` `provider` `server` | | `coder_ai_gateway_interceptions_duration_seconds` | histogram | The total duration of intercepted requests, in seconds. The majority of this time will be the upstream processing of the request. AI Gateway has no control over upstream processing time, so it's just an illustrative metric. | `model` `provider` | | `coder_ai_gateway_interceptions_inflight` | gauge | The number of intercepted requests which are being processed. | `model` `provider` `route` | | `coder_ai_gateway_interceptions_total` | counter | The count of intercepted requests. | `client` `initiator_id` `method` `model` `provider` `route` `status` | | `coder_ai_gateway_key_pool_exhaustions_total` | counter | The number of times the key pool was exhausted with no usable key (outcome: rate\_limited, auth\_failed). | `outcome` `provider` | | `coder_ai_gateway_key_pool_failover_attempts` | histogram | The number of keys attempted before success or exhaustion, per interception for bridged requests and per request for passthrough requests. | `provider` | | `coder_ai_gateway_key_pool_state` | gauge | The number of keys currently in each state (state: valid, temporary, permanent). | `provider` `state` | | `coder_ai_gateway_key_pool_state_transitions_total` | counter | The number of API key state transitions during failover (reason: rate\_limited, unauthorized). | `provider` `reason` | | `coder_ai_gateway_non_injected_tool_selections_total` | counter | The number of times an AI model selected a tool to be invoked by the client. | `model` `name` `provider` | | `coder_ai_gateway_passthrough_total` | counter | The count of requests which were not intercepted but passed through to the upstream. | `method` `provider` `route` | | `coder_ai_gateway_prompts_total` | counter | The number of prompts issued by users (initiators). | `client` `initiator_id` `model` `provider` | | `coder_ai_gateway_provider_info` | gauge | One series per configured AI provider. Value is always 1; the status label (enabled, disabled, error) carries the alertable signal. | `provider_name` `provider_type` `status` | | `coder_ai_gateway_providers_last_reload_success_timestamp_seconds` | gauge | Unix timestamp of the last provider reload that successfully refreshed the pool. A gap against the providers\_last\_reload\_timestamp\_seconds gauge means the loop is firing but the refresh function is failing. | | | `coder_ai_gateway_providers_last_reload_timestamp_seconds` | gauge | Unix timestamp of the last provider reload attempt, success or failure. | | | `coder_ai_gateway_proxy_connect_sessions_total` | counter | Total number of CONNECT sessions established. | `type` | | `coder_ai_gateway_proxy_inflight_mitm_requests` | gauge | Number of MITM requests currently being processed. | `provider` | | `coder_ai_gateway_proxy_mitm_requests_total` | counter | Total number of MITM requests handled by the proxy. | `provider` | | `coder_ai_gateway_proxy_mitm_responses_total` | counter | Total number of MITM responses by HTTP status code class. | `code` `provider` | | `coder_ai_gateway_proxy_provider_info` | gauge | One series per configured AI provider. Value is always 1; the status label (enabled, disabled, error, proxy\_excluded) carries the alertable signal. | `provider_name` `provider_type` `status` | | `coder_ai_gateway_proxy_providers_last_reload_success_timestamp_seconds` | gauge | Unix timestamp of the last provider reload that successfully refreshed the router. A gap against the providers\_last\_reload\_timestamp\_seconds gauge means the loop is firing but the refresh function is failing. | | | `coder_ai_gateway_proxy_providers_last_reload_timestamp_seconds` | gauge | Unix timestamp of the last provider reload attempt, success or failure. | | | `coder_ai_gateway_tokens_total` | counter | The number of tokens used by intercepted requests. | `client` `initiator_id` `model` `provider` `type` | | `coder_derp_server_accepts_total` | counter | Total DERP connections accepted. | | | `coder_derp_server_average_queue_duration_ms` | gauge | Average queue duration in milliseconds. | | | `coder_derp_server_bytes_received_total` | counter | Total bytes received. | | | `coder_derp_server_bytes_sent_total` | counter | Total bytes sent. | | | `coder_derp_server_clients` | gauge | Total clients (local + remote). | | | `coder_derp_server_clients_local` | gauge | Local clients. | | | `coder_derp_server_clients_remote` | gauge | Remote (mesh) clients. | | | `coder_derp_server_connections` | gauge | Current DERP connections. | | | `coder_derp_server_got_ping_total` | counter | Total pings received. | | | `coder_derp_server_home_connections` | gauge | Current home DERP connections. | | | `coder_derp_server_home_moves_in_total` | counter | Total home moves in. | | | `coder_derp_server_home_moves_out_total` | counter | Total home moves out. | | | `coder_derp_server_packets_dropped_reason_total` | counter | Packets dropped by reason. | `reason` | | `coder_derp_server_packets_dropped_total` | counter | Total packets dropped. | | | `coder_derp_server_packets_dropped_type_total` | counter | Packets dropped by type. | `type` | | `coder_derp_server_packets_forwarded_in_total` | counter | Total packets forwarded in from mesh peers. | | | `coder_derp_server_packets_forwarded_out_total` | counter | Total packets forwarded out to mesh peers. | | | `coder_derp_server_packets_received_kind_total` | counter | Packets received by kind. | `kind` | | `coder_derp_server_packets_received_total` | counter | Total packets received. | | | `coder_derp_server_packets_sent_total` | counter | Total packets sent. | | | `coder_derp_server_peer_gone_disconnected_total` | counter | Total peer gone (disconnected) frames sent. | | | `coder_derp_server_peer_gone_not_here_total` | counter | Total peer gone (not here) frames sent. | | | `coder_derp_server_sent_pong_total` | counter | Total pongs sent. | | | `coder_derp_server_unknown_frames_total` | counter | Total unknown frames received. | | | `coder_derp_server_watchers` | gauge | Current watchers. | | | `coder_nats_pubsub_connected` | gauge | Whether we are connected (1) or not connected (0) to nats | | | `coder_nats_pubsub_current_events` | gauge | The current number of pubsub event channels listened for | | | `coder_nats_pubsub_current_subscribers` | gauge | The current number of active pubsub subscribers | | | `coder_nats_pubsub_disconnections_total` | counter | Total number of times we disconnected unexpectedly from nats | | | `coder_nats_pubsub_latency_measure_errs_total` | counter | The number of pubsub latency measurement failures | | | `coder_nats_pubsub_latency_measures_total` | counter | The number of pubsub latency measurements | | | `coder_nats_pubsub_messages_total` | counter | Total number of messages received from nats | `size` | | `coder_nats_pubsub_published_bytes_total` | counter | Total number of bytes successfully published across all publishes | | | `coder_nats_pubsub_publishes_total` | counter | Total number of calls to Publish | `success` | | `coder_nats_pubsub_receive_latency_seconds` | gauge | The time taken to receive a message from a pubsub event channel | | | `coder_nats_pubsub_received_bytes_total` | counter | Total number of bytes received across all messages | | | `coder_nats_pubsub_send_latency_seconds` | gauge | The time taken to send a message into a pubsub event channel | | | `coder_nats_pubsub_subscribes_total` | counter | Total number of calls to Subscribe/SubscribeWithErr | `success` | | `coder_pubsub_connected` | gauge | Whether we are connected (1) or not connected (0) to postgres | | | `coder_pubsub_current_events` | gauge | The current number of pubsub event channels listened for | | | `coder_pubsub_current_subscribers` | gauge | The current number of active pubsub subscribers | | | `coder_pubsub_disconnections_total` | counter | Total number of times we disconnected unexpectedly from postgres | | | `coder_pubsub_latency_measure_errs_total` | counter | The number of pubsub latency measurement failures | | | `coder_pubsub_latency_measures_total` | counter | The number of pubsub latency measurements | | | `coder_pubsub_messages_total` | counter | Total number of messages received from postgres | `size` | | `coder_pubsub_published_bytes_total` | counter | Total number of bytes successfully published across all publishes | | | `coder_pubsub_publishes_total` | counter | Total number of calls to Publish | `success` | | `coder_pubsub_receive_latency_seconds` | gauge | The time taken to receive a message from a pubsub event channel | | | `coder_pubsub_received_bytes_total` | counter | Total number of bytes received across all messages | | | `coder_pubsub_send_latency_seconds` | gauge | The time taken to send a message into a pubsub event channel | | | `coder_pubsub_subscribes_total` | counter | Total number of calls to Subscribe/SubscribeWithErr | `success` | | `coder_servertailnet_connections_total` | counter | Total number of TCP connections made to workspace agents. | `network` | | `coder_servertailnet_open_connections` | gauge | Total number of TCP connections currently open to workspace agents. | `network` | | `coderd_agentapi_metadata_batch_size` | histogram | Total number of metadata entries in each batch, updated before flushes. | | | `coderd_agentapi_metadata_batch_utilization` | histogram | Number of metadata keys per agent in each batch, updated before flushes. | | | `coderd_agentapi_metadata_batches_total` | counter | Total number of metadata batches flushed. | `reason` | | `coderd_agentapi_metadata_dropped_keys_total` | counter | Total number of metadata keys dropped due to capacity limits. | | | `coderd_agentapi_metadata_flush_duration_seconds` | histogram | Time taken to flush metadata batch to database and pubsub. | `reason` | | `coderd_agentapi_metadata_flushed_total` | counter | Total number of unique metadatas flushed. | | | `coderd_agentapi_metadata_publish_errors_total` | counter | Total number of metadata batch pubsub publish calls that have resulted in an error. | | | `coderd_agents_apps` | gauge | Agent applications with statuses. | `agent_name` `app_name` `health` `username` `workspace_name` | | `coderd_agents_connection_latencies_seconds` | gauge | Agent connection latencies in seconds. | `agent_name` `derp_region` `preferred` `username` `workspace_name` | | `coderd_agents_connections` | gauge | Agent connections with statuses. | `agent_name` `lifecycle_state` `status` `tailnet_node` `username` `workspace_name` | | `coderd_agents_first_connection_seconds` | histogram | Duration from agent creation to first connection in seconds. | `agent_name` `template_name` | | `coderd_agents_up` | gauge | The number of active agents per workspace. | `template_name` `template_version` `username` `workspace_name` | | `coderd_agentstats_connection_count` | gauge | The number of established connections by agent | `agent_name` `username` `workspace_name` | | `coderd_agentstats_connection_median_latency_seconds` | gauge | The median agent connection latency | `agent_name` `username` `workspace_name` | | `coderd_agentstats_currently_reachable_peers` | gauge | The number of peers (e.g. clients) that are currently reachable over the encrypted network. | `agent_name` `connection_type` `template_name` `username` `workspace_name` | | `coderd_agentstats_rx_bytes` | gauge | Agent Rx bytes | `agent_name` `username` `workspace_name` | | `coderd_agentstats_session_count_jetbrains` | gauge | The number of session established by JetBrains | `agent_name` `username` `workspace_name` | | `coderd_agentstats_session_count_reconnecting_pty` | gauge | The number of session established by reconnecting PTY | `agent_name` `username` `workspace_name` | | `coderd_agentstats_session_count_ssh` | gauge | The number of session established by SSH | `agent_name` `username` `workspace_name` | | `coderd_agentstats_session_count_vscode` | gauge | The number of session established by VSCode | `agent_name` `username` `workspace_name` | | `coderd_agentstats_startup_script_seconds` | gauge | The number of seconds the startup script took to execute. | `agent_name` `success` `template_name` `username` `workspace_name` | | `coderd_agentstats_tx_bytes` | gauge | Agent Tx bytes | `agent_name` `username` `workspace_name` | | `coderd_api_active_users_duration_hour` | gauge | The number of users that have been active within the last hour. | | | `coderd_api_concurrent_requests` | gauge | The number of concurrent API requests. | `method` `path` | | `coderd_api_concurrent_websockets` | gauge | The total number of concurrent API websockets. | `path` | | `coderd_api_request_latencies_seconds` | histogram | Latency distribution of requests in seconds. | `method` `path` | | `coderd_api_requests_processed_total` | counter | The total number of processed API requests | `code` `method` `path` | | `coderd_api_requests_too_large_total` | counter | The total number of API requests answered 413, by the reason they were rejected. A sustained rate of reason="request\_body" on one route is more often a limit set too tight for a legitimate payload than an attempt to exhaust memory. reason="other" counts 413s raised for reasons unrelated to the size of the request body, such as agent log storage overflow or an archive that is too large once expanded. | `method` `path` `reason` | | `coderd_api_total_user_count` | gauge | The total number of registered users, partitioned by status. | `status` | | `coderd_api_websocket_durations_seconds` | histogram | Websocket duration distribution of requests in seconds. | `path` | | `coderd_api_websocket_probes_total` | counter | WebSocket liveness probe outcomes by route. Compare rate(...\{result="ok"}\[1m]) against coderd\_api\_concurrent\_websockets to detect unresponsive WebSocket connections. | `path` `result` | | `coderd_api_workspace_latest_build` | gauge | The current number of workspace builds by status for all non-deleted workspaces. | `status` | | `coderd_authz_authorize_duration_seconds` | histogram | Duration of the 'Authorize' call in seconds. Only counts calls that succeed. | `allowed` | | `coderd_authz_prepare_authorize_duration_seconds` | histogram | Duration of the 'PrepareAuthorize' call in seconds. | | | `coderd_build_info` | gauge | Describes the current build/version of the Coder server. Value is always 1. | `revision` `version` | | `coderd_chat_auto_archive_records_archived_total` | counter | Total number of chats archived by the auto-archive job (counting both roots and cascaded children). | | | `coderd_chatd_agents_active` | gauge | Deployment-wide number of chats holding a concurrent-agent capacity slot. Every replica reports the same database-derived value; aggregate with max, not sum. | `pool` | | `coderd_chatd_agents_queued_for_capacity` | gauge | Deployment-wide number of chats waiting for a concurrent-agent capacity slot. Every replica reports the same database-derived value; aggregate with max, not sum. | `pool` | | `coderd_chatd_chats` | gauge | Number of chats being processed, by state. | `state` | | `coderd_chatd_compaction_total` | counter | Total compaction outcomes (only recorded when compaction was triggered or failed). | `model` `provider` `result` | | `coderd_chatd_find_tools_activations_total` | counter | Total deferred tool activations returned by find\_tools. | | | `coderd_chatd_find_tools_calls_total` | counter | Total find\_tools calls. | | | `coderd_chatd_find_tools_empty_total` | counter | Total find\_tools calls with no matches. | | | `coderd_chatd_find_tools_match_count` | histogram | Number of matches returned by find\_tools calls. | | | `coderd_chatd_hook_context_size_bytes` | histogram | Lifecycle hook model context response size in bytes. | `event` | | `coderd_chatd_hook_decisions_total` | counter | Total lifecycle hook permission decisions by event and decision. | `decision` `event` | | `coderd_chatd_hook_dispatch_seconds` | histogram | Lifecycle hook dispatch duration in seconds. | `event` | | `coderd_chatd_hook_dispatches_total` | counter | Total lifecycle hook dispatches by event and result. | `event` `result` | | `coderd_chatd_hook_input_overrides_total` | counter | Total lifecycle hook input overrides by event. | `event` | | `coderd_chatd_message_count` | histogram | Number of messages in the prompt per LLM request. | `model` `provider` | | `coderd_chatd_prompt_size_bytes` | histogram | Estimated byte size of the prompt per LLM request. | `model` `provider` | | `coderd_chatd_steps_total` | counter | Total agentic loop steps across all chats. | `model` `provider` | | `coderd_chatd_stream_buffer_dropped_total` | counter | Number of chat stream buffer events dropped due to the per-chat buffer cap. | | | `coderd_chatd_stream_retries_total` | counter | Total LLM stream retries. | `kind` `model` `provider` | | `coderd_chatd_tool_errors_total` | counter | Total tool calls that returned an error result. | `model` `provider` `tool_name` | | `coderd_chatd_tool_result_size_bytes` | histogram | Size in bytes of each tool execution result. | `model` `provider` `tool_name` | | `coderd_chatd_tool_result_truncated_total` | counter | Total tool results truncated to fit the model context window. | `model` `provider` `tool_name` | | `coderd_chatd_ttft_seconds` | histogram | Time-to-first-token: wall time from LLM request to first streamed chunk. | `model` `provider` | | `coderd_db_query_counts_total` | counter | Total number of queries labelled by HTTP route, method, and query name. | `method` `query` `route` | | `coderd_db_query_latencies_seconds` | histogram | Latency distribution of queries in seconds. | `query` | | `coderd_db_tx_duration_seconds` | histogram | Duration of transactions in seconds. | `success` `tx_id` | | `coderd_db_tx_executions_count` | counter | Total count of transactions executed. 'retries' is expected to be 0 for a successful transaction. | `retries` `success` `tx_id` | | `coderd_dbpurge_chat_search_rows_backfilled_total` | counter | Total number of chat message rows whose search\_tsv was backfilled. | | | `coderd_dbpurge_iteration_duration_seconds` | histogram | Duration of each dbpurge iteration in seconds. | `success` | | `coderd_dbpurge_records_purged_total` | counter | Total number of records purged by type. | `record_type` | | `coderd_experiments` | gauge | Indicates whether each experiment is enabled (1) or not (0) | `experiment` | | `coderd_insights_applications_usage_seconds` | gauge | The application usage per template. | `application_name` `organization_name` `slug` `template_name` | | `coderd_insights_parameters` | gauge | The parameter usage per template. | `organization_name` `parameter_name` `parameter_type` `parameter_value` `template_name` | | `coderd_insights_templates_active_users` | gauge | The number of active users of the template. | `organization_name` `template_name` | | `coderd_license_active_users` | gauge | The number of active users. | | | `coderd_license_errors` | gauge | The number of active license errors. | | | `coderd_license_limit_users` | gauge | The user seats limit based on the active Coder license. | | | `coderd_license_user_limit_enabled` | gauge | Returns 1 if the current license enforces the user limit. | | | `coderd_license_warnings` | gauge | The number of active license warnings. | | | `coderd_lifecycle_autobuild_execution_duration_seconds` | histogram | Duration of each autobuild execution. | | | `coderd_notifications_dispatcher_send_seconds` | histogram | The time taken to dispatch notifications. | `method` | | `coderd_notifications_inflight_dispatches` | gauge | The number of dispatch attempts which are currently in progress. | `method` `notification_template_id` | | `coderd_notifications_pending_updates` | gauge | The number of dispatch attempt results waiting to be flushed to the store. | | | `coderd_notifications_queued_seconds` | histogram | The time elapsed between a notification being enqueued in the store and retrieved for dispatching (measures the latency of the notifications system). This should generally be within CODER\_NOTIFICATIONS\_FETCH\_INTERVAL seconds; higher values for a sustained period indicates delayed processing and CODER\_NOTIFICATIONS\_LEASE\_COUNT can be increased to accommodate this. | `method` | | `coderd_notifications_retry_count` | counter | The count of notification dispatch retry attempts. | `method` `notification_template_id` | | `coderd_notifications_synced_updates_total` | counter | The number of dispatch attempt results flushed to the store. | | | `coderd_oauth2_external_requests_rate_limit` | gauge | The total number of allowed requests per interval. | `name` `resource` | | `coderd_oauth2_external_requests_rate_limit_next_reset_unix` | gauge | Unix timestamp for when the next interval starts | `name` `resource` | | `coderd_oauth2_external_requests_rate_limit_remaining` | gauge | The remaining number of allowed requests in this interval. | `name` `resource` | | `coderd_oauth2_external_requests_rate_limit_reset_in_seconds` | gauge | Seconds until the next interval | `name` `resource` | | `coderd_oauth2_external_requests_rate_limit_used` | gauge | The number of requests made in this interval. | `name` `resource` | | `coderd_oauth2_external_requests_rate_limited_total` | counter | The total number of api calls to external oauth2 providers that returned a rate-limited response (a 429, or a 403 with rate-limit headers). | `name` `source` `status_code` | | `coderd_oauth2_external_requests_total` | counter | The total number of api calls made to external oauth2 providers. 'status\_code' will be 0 if the request failed with no response. | `name` `source` `status_code` | | `coderd_open_file_refs_current` | gauge | The count of file references currently open in the file cache. Multiple references can be held for the same file. | | | `coderd_open_file_refs_total` | counter | The total number of file references ever opened in the file cache. The 'hit' label indicates if the file was loaded from the cache. | `hit` | | `coderd_open_files_current` | gauge | The count of unique files currently open in the file cache. | | | `coderd_open_files_size_bytes_current` | gauge | The current amount of memory of all files currently open in the file cache. | | | `coderd_open_files_size_bytes_total` | counter | The total amount of memory ever opened in the file cache. This number never decrements. | | | `coderd_open_files_total` | counter | The total count of unique files ever opened in the file cache. | | | `coderd_prebuilds_reconciliation_duration_seconds` | histogram | Duration of each prebuilds reconciliation cycle. | | | `coderd_prebuilt_workspace_claim_duration_seconds` | histogram | Time to claim a prebuilt workspace by organization, template, and preset. | `organization_name` `preset_name` `template_name` | | `coderd_prebuilt_workspaces_claimed_total` | counter | Total number of prebuilt workspaces which were claimed by users. Claiming refers to creating a workspace with a preset selected for which eligible prebuilt workspaces are available and one is reassigned to a user. | `organization_name` `preset_name` `template_name` | | `coderd_prebuilt_workspaces_created_total` | counter | Total number of prebuilt workspaces that have been created to meet the desired instance count of each template preset. | `organization_name` `preset_name` `template_name` | | `coderd_prebuilt_workspaces_desired` | gauge | Target number of prebuilt workspaces that should be available for each template preset. | `organization_name` `preset_name` `template_name` | | `coderd_prebuilt_workspaces_eligible` | gauge | Current number of prebuilt workspaces that are eligible to be claimed by users. These are workspaces that have completed their build process with their agent reporting 'ready' status. | `organization_name` `preset_name` `template_name` | | `coderd_prebuilt_workspaces_failed_total` | counter | Total number of prebuilt workspaces that failed to build. | `organization_name` `preset_name` `template_name` | | `coderd_prebuilt_workspaces_metrics_last_updated` | gauge | The unix timestamp when the metrics related to prebuilt workspaces were last updated; these metrics are cached. | | | `coderd_prebuilt_workspaces_preset_hard_limited` | gauge | Indicates whether a given preset has reached the hard failure limit (1 = hard-limited). Metric is omitted otherwise. | `organization_name` `preset_name` `template_name` | | `coderd_prebuilt_workspaces_preset_validation_failed` | gauge | Indicates whether a given preset has validation failures (1 = validation failed). Metric is omitted otherwise. | `organization_name` `preset_name` `template_name` | | `coderd_prebuilt_workspaces_reconciliation_paused` | gauge | Indicates whether prebuilds reconciliation is currently paused (1 = paused, 0 = not paused). | | | `coderd_prebuilt_workspaces_resource_replacements_total` | counter | Total number of prebuilt workspaces whose resource(s) got replaced upon being claimed. In Terraform, drift on immutable attributes results in resource replacement. This represents a worst-case scenario for prebuilt workspaces because the pre-provisioned resource would have been recreated when claiming, thus obviating the point of pre-provisioning. See [https://coder.com/docs/admin/templates/extending-templates/prebuilt-workspaces#preventing-resource-replacement](https://coder.com/docs/admin/templates/extending-templates/prebuilt-workspaces#preventing-resource-replacement) | `organization_name` `preset_name` `template_name` | | `coderd_prebuilt_workspaces_running` | gauge | Current number of prebuilt workspaces that are in a running state. These workspaces have started successfully but may not yet be claimable by users (see coderd\_prebuilt\_workspaces\_eligible). | `organization_name` `preset_name` `template_name` | | `coderd_prometheusmetrics_agents_execution_seconds` | histogram | Histogram for duration of agents metrics collection in seconds. | | | `coderd_prometheusmetrics_agentstats_execution_seconds` | histogram | Histogram for duration of agent stats metrics collection in seconds. | | | `coderd_prometheusmetrics_metrics_aggregator_execution_cleanup_seconds` | histogram | Histogram for duration of metrics aggregator cleanup in seconds. | | | `coderd_prometheusmetrics_metrics_aggregator_execution_update_seconds` | histogram | Histogram for duration of metrics aggregator update in seconds. | | | `coderd_prometheusmetrics_metrics_aggregator_store_size` | gauge | The number of metrics stored in the aggregator | | | `coderd_provisioner_job_queue_wait_seconds` | histogram | Time from job creation to acquisition by a provisioner daemon. | `build_reason` `job_type` `provisioner_type` `transition` | | `coderd_provisionerd_job_timings_seconds` | histogram | The provisioner job time duration in seconds. | `provisioner` `status` | | `coderd_provisionerd_jobs_current` | gauge | The number of currently running provisioner jobs. | `provisioner` | | `coderd_provisionerd_num_daemons` | gauge | The number of provisioner daemons. | | | `coderd_provisionerd_workspace_build_timings_seconds` | histogram | The time taken for a workspace to build. | `status` `template_name` `template_version` `workspace_transition` | | `coderd_proxyhealth_health_check_duration_seconds` | histogram | Histogram for duration of proxy health collection in seconds. | | | `coderd_proxyhealth_health_check_results` | gauge | This endpoint returns a number to indicate the health status. -3 (unknown), -2 (Unreachable), -1 (Unhealthy), 0 (Unregistered), 1 (Healthy) | `proxy_id` | | `coderd_template_workspace_build_duration_seconds` | histogram | Duration from workspace build creation to agent ready, by template. | `is_prebuild` `organization_name` `status` `template_name` `transition` | | `coderd_workspace_builds_enqueued_total` | counter | Total number of workspace build enqueue attempts. | `build_reason` `provisioner_type` `status` `transition` | | `coderd_workspace_builds_total` | counter | The number of workspaces started, updated, or deleted. | `status` `template_name` `template_version` `workspace_name` `workspace_owner` `workspace_transition` | | `coderd_workspace_creation_duration_seconds` | histogram | Time to create a workspace by organization, template, preset, and type (regular or prebuild). | `organization_name` `preset_name` `template_name` `type` | | `coderd_workspace_creation_total` | counter | Total regular (non-prebuilt) workspace creations by organization, template, and preset. | `organization_name` `preset_name` `template_name` | | `coderd_workspace_latest_build_status` | gauge | The current workspace statuses by template, transition, and owner for all non-deleted workspaces. | `status` `template_name` `template_version` `workspace_owner` `workspace_transition` | | `go_gc_duration_seconds` | summary | A summary of the pause duration of garbage collection cycles. | | | `go_goroutines` | gauge | Number of goroutines that currently exist. | | | `go_info` | gauge | Information about the Go environment. | `version` | | `go_memstats_alloc_bytes` | gauge | Number of bytes allocated and still in use. | | | `go_memstats_alloc_bytes_total` | counter | Total number of bytes allocated, even if freed. | | | `go_memstats_buck_hash_sys_bytes` | gauge | Number of bytes used by the profiling bucket hash table. | | | `go_memstats_frees_total` | counter | Total number of frees. | | | `go_memstats_gc_sys_bytes` | gauge | Number of bytes used for garbage collection system metadata. | | | `go_memstats_heap_alloc_bytes` | gauge | Number of heap bytes allocated and still in use. | | | `go_memstats_heap_idle_bytes` | gauge | Number of heap bytes waiting to be used. | | | `go_memstats_heap_inuse_bytes` | gauge | Number of heap bytes that are in use. | | | `go_memstats_heap_objects` | gauge | Number of allocated objects. | | | `go_memstats_heap_released_bytes` | gauge | Number of heap bytes released to OS. | | | `go_memstats_heap_sys_bytes` | gauge | Number of heap bytes obtained from system. | | | `go_memstats_last_gc_time_seconds` | gauge | Number of seconds since 1970 of last garbage collection. | | | `go_memstats_lookups_total` | counter | Total number of pointer lookups. | | | `go_memstats_mallocs_total` | counter | Total number of mallocs. | | | `go_memstats_mcache_inuse_bytes` | gauge | Number of bytes in use by mcache structures. | | | `go_memstats_mcache_sys_bytes` | gauge | Number of bytes used for mcache structures obtained from system. | | | `go_memstats_mspan_inuse_bytes` | gauge | Number of bytes in use by mspan structures. | | | `go_memstats_mspan_sys_bytes` | gauge | Number of bytes used for mspan structures obtained from system. | | | `go_memstats_next_gc_bytes` | gauge | Number of heap bytes when next garbage collection will take place. | | | `go_memstats_other_sys_bytes` | gauge | Number of bytes used for other system allocations. | | | `go_memstats_stack_inuse_bytes` | gauge | Number of bytes in use by the stack allocator. | | | `go_memstats_stack_sys_bytes` | gauge | Number of bytes obtained from system for stack allocator. | | | `go_memstats_sys_bytes` | gauge | Number of bytes obtained from system. | | | `go_threads` | gauge | Number of OS threads created. | | | `process_cpu_seconds_total` | counter | Total user and system CPU time spent in seconds. | | | `process_max_fds` | gauge | Maximum number of open file descriptors. | | | `process_open_fds` | gauge | Number of open file descriptors. | | | `process_resident_memory_bytes` | gauge | Resident memory size in bytes. | | | `process_start_time_seconds` | gauge | Start time of the process since unix epoch in seconds. | | | `process_virtual_memory_bytes` | gauge | Virtual memory size in bytes. | | | `process_virtual_memory_max_bytes` | gauge | Maximum amount of virtual memory available in bytes. | | | `promhttp_metric_handler_requests_in_flight` | gauge | Current number of scrapes being served. | | | `promhttp_metric_handler_requests_total` | counter | Total number of scrapes by HTTP status code. | `code` | ### Note on Prometheus native histogram support [#note-on-prometheus-native-histogram-support] The following metrics support native histograms: * `coderd_workspace_creation_duration_seconds` * `coderd_prebuilt_workspace_claim_duration_seconds` * `coderd_template_workspace_build_duration_seconds` Native histograms are an **experimental** Prometheus feature that removes the need to predefine bucket boundaries and allows higher-resolution buckets that adapt to deployment characteristics. Whether a metric is exposed as classic or native depends entirely on the Prometheus server configuration (see [Prometheus docs](https://prometheus.io/docs/specs/native_histograms/) for details): * If native histograms are enabled, Prometheus ingests the high-resolution histogram. * If not, it falls back to the predefined buckets. ⚠️ Important: classic and native histograms cannot be aggregated together. If Prometheus is switched from classic to native at a certain point in time, dashboards may need to account for that transition. For this reason, it’s recommended to follow [Prometheus’ migration guidelines](https://prometheus.io/docs/specs/native_histograms/#migration-considerations) when moving from classic to native histograms.