Taruj Goyal

Senior ML/Infrastructure Engineer at Abridge

Sam Robertson

Senior Machine Learning Engineer at Abridge

<p>Abridge is one of the fastest-growing AI companies in healthcare. This year, the enterprise-grade AI platform for clinical conversations is projected to support more than 80 million patient-clinician conversations across 250 of the largest and most complex health systems in the U.S. With that expansion comes a familiar engineering challenge: the infrastructure that supported a small initial team of developers doesn’t scale to meet skyrocketing demand. </p>

<p>More health systems means more feature requirements. More features means more engineers. More engineers means more machines, more configurations, more drift, and more time that ML Infra teams have to spend keeping the lights on instead of building. At a certain scale, developer tooling stops being a convenience and starts being a constraint on how fast the business can move.</p>

<p>This is the story of how Abridge rearchitected its developer infrastructure using Coder and transformed developer onboarding time from days to minutes. </p>

<p><img src="https://www.datocms-assets.com/19109/1774555038-blog_abridge-graphic.png" alt=""></p>

<h2><strong>The scaling problem is a compounding problem</strong></h2>

<p>To build for the complex realities of healthcare, Abridge relies on several developer archetypes working in parallel: clinical scientists processing sensitive data, ML researchers training and evaluating models on GPU clusters, product engineers building APIs and inference services, and PMs prototyping internal dashboards and prototypes. Each archetype has different resource requirements, different compliance constraints, and a different tolerance for infrastructure complexity.</p>

<p>As the team grows, the friction compounds. An ML Infra engineer debugging a colleague&#39;s environment is an ML Infra engineer not building infrastructure. A clinical scientist blocked on cloud access is a clinical scientist not building data intake workflows. The compounding effect is the real problem: every hour of environment overhead is an hour not spent on the 100+ data intake workflows and 50 ML models that are core to the product. </p>

<p><strong>Infrastructure as code, all the way down</strong></p>

<p>The foundation of the current setup is a Terraform-based workspace system deployed on GCP. Everything that defines how a workspace behaves lives in version control: compute sizing, network policy, secrets management, compliance controls, lifecycle rules, and cost tracking. No one needs admin access to a UI to change how workspaces behave. Changes go through PRs and CI like everything else.</p>

<p><img src="https://www.datocms-assets.com/19109/1774555413-blog_abridge-architecture-2.png" alt="">
<em>The full system architecture: developer self-service on the left, GitOps-driven control plane on the right. Every workspace type that flows through Terraform applies to the Kubernetes cluster with no manual steps.</em></p>

<p>We organized the architecture into three layers. Shared modules form the base, providing standardized resource definitions and reusable utilities that every template inherits. Templates sit in the middle, encoding workspace behavior for each developer archetype. The control plane sits at the top, managing the Coder server itself via Helm with per-environment value overrides for dev, staging, and production.</p>

<p><img src="https://www.datocms-assets.com/19109/1774555816-blog_abridge-architecture-3.png" alt="">
 <em>The three-layer architecture: shared modules define resource parameters and utilities; templates consume them to define workspace types; the control plane manages secrets, auth, networking, and per-environment config via Helm and Istio.</em></p>

<p>Abridge maintains seven Terraform-based templates covering the full range of developer needs:</p>

<table><thead>
<tr>
<th>Template</th>
<th>Use Case</th>
</tr>
</thead><tbody>
<tr>
<td><code>cpu-k8s</code></td>
<td>Standard dev workspaces</td>
</tr>
<tr>
<td><code>clinician-k8s</code></td>
<td>CPU workspaces with Phoenix observability sidecar</td>
</tr>
<tr>
<td><code>gpu-k8s</code></td>
<td>ML/AI workspaces with L4/H100 GPU access</td>
</tr>
<tr>
<td><code>phi-gpu-k8s</code></td>
<td>PHI-compliant, air-gapped GPU workspaces</td>
</tr>
<tr>
<td><code>gcp-vm-modular</code></td>
<td>GCP VMs with deep learning images</td>
</tr>
<tr>
<td><code>skypilot-k8s</code></td>
<td>Multi-cloud job orchestration</td>
</tr>
<tr>
<td><code>soap-dashboard-k8s</code></td>
<td>Dashboard dev for PMs</td>
</tr>
</tbody></table>

<p><img src="https://www.datocms-assets.com/19109/1774555816-blog_abridge-architecture-5.png" alt=""></p>

<p><em>Workspace templates grouped by access tier: restricted PHI workspaces with zero external egress, GPU workspaces with L4/H100 access, and standard workspaces for dev, clinical science, multi-cloud orchestration, and dashboard work.</em></p>

<p>Each template exposes mutable parameters: CPU, memory, disk, GPU type and count, repo URL, and IDE preference. Developers tune their own environments within guardrails set by the template. No admin intervention required. <code>/modules/resources/</code> defines standardized parameter ranges and cost definitions. <code>/modules/utilities/</code> handles reusable setup tasks: git clone, IDE configuration, GCS bucket mounting, and logging. Changing a default in one place propagates everywhere.</p>

<p><img src="https://www.datocms-assets.com/19109/1774556195-image4.png" alt=""></p>

<h2><strong>Compliance encoded, not enforced</strong></h2>

<p>For workspaces that handle patient data, compliance requirements aren&#39;t a checklist in a UI. They&#39;re structural.</p>

<p>Abridge designed the <code>phi-gpu-k8s</code> template to enforce HIPAA-aligned constraints at the network layer using Istio ServiceEntry and Sidecar configs: zero external egress by default, with allow-listed endpoints only (PyPI, GCS, BigQuery, internal services). File transfer is disabled at the agent level. Browser-only access is enforced—no port forwarding, no direct shell access. These aren&#39;t settings a user can accidentally misconfigure. They&#39;re baked into the network policy definitions committed to version control.</p>

<p>Abridge also built a clear separation between ephemeral and persistent data models into the template design itself. Category I data persists on workspace volumes across restarts. Category II workspaces run ephemerally—no data written to disk, no state retained between sessions. The distinction is enforced by how the templates are architected, not by policy documentation that developers have to remember to follow. When your product handles sensitive clinical conversations, the difference between &quot;policy says don&#39;t do this&quot; and &quot;the system makes it impossible&quot; is significant.</p>

<h2><strong>CI/CD for infrastructure changes</strong></h2>

<p>Template updates don&#39;t require anyone to log into a console. Abridge built two GitHub Actions workflows to handle the full template lifecycle. The deploy workflow fires on merge to main or manual dispatch, running <code>terraform init / validate / plan / apply</code> per template and versioning each by commit SHA. The test workflow fires on every PR touching templates or modules, building a test matrix and running parallel tests against scenario configs in <code>tests/templates/*.json</code> before anything reaches production.</p>

<p><img src="https://www.datocms-assets.com/19109/1774555816-blog_abridge-architecture-4.png" alt="">
<em>Two GitHub Actions workflows handle all template lifecycle management: the deploy workflow fires on merge to main or manual dispatch; the test workflow fires on every PR touching templates or modules, running parallel tests against scenario configs before anything reaches production.</em></p>

<p>ML infra engineers can safely iterate on templates in the dev environment and validate against staging before anything touches production users. The same rigor Abridge applies to application code applies to the infrastructure that runs it. That consistency matters when your infrastructure changes can affect GPU workloads running sensitive model training jobs.</p>

<h2><strong>Self-service onboarding that actually works</strong></h2>

<p>The developer flow today:</p>

<ol>
<li>Added to the appropriate GitHub org team<br></li>
<li>VPN connection<br></li>
<li>GitHub SSO into Coder<br></li>
<li>Template selection and parameter tuning<br></li>
<li>Workspace provisions automatically with repo cloned, tooling configured, and cloud access authenticated</li>
</ol>

<p><img src="https://www.datocms-assets.com/19109/1774555816-blog_abridge-architecture-1.png" alt="">
<em>End-to-end workspace provisioning: GitHub OAuth with org and team gating, template selection with user-tunable parameters, Terraform-driven PVC and Deployment creation in Kubernetes, and a workspace that arrives with git repo cloned, Python and uv installed, and metrics reporting active.</em></p>

<p>Every repository includes a <code>coder_setup.sh</code> that handles virtualenv creation, dependency installation from private registries, service-specific tooling, and pre-commit hooks. Abridge made this a deliberate architectural choice: templates stay minimal and stable, while repos own their own setup logic. A new ML scientist who joined recently had their first training job running on day one -- same environment as the rest of the team, no specialized help required.</p>

<h2><strong>GPU cost as an engineering problem</strong></h2>

<p>GPU waste is an expensive problem at any ML-focused company, and it tends to be a social problem as much as a technical one. Jobs finish but instances stay live.</p>

<p>Abridge addressed this structurally through the template design. Auto-stop policies are configured per template so idle workspaces shut down automatically. Compute allocations are right-sized to workload type at the template level rather than left to individual developers to decide. Resource limits are consistent and predictable across the fleet because they come from the same shared modules. The result was immediate: significant decommissioning of stray cloud VMs and a material reduction in GPU spend, without developers feeling constrained because the system handles it automatically.</p>

<p>Abridge also runs three independent environments—dev, staging, and production—each with independent access controls, quotas, and policies. The separation gives ML Infra engineers room to test infrastructure changes safely without touching production workloads, which at our GPU spend levels is not a theoretical concern.</p>

<h2><strong>What Abridge learned building this</strong></h2>

<p>The decisions that made the biggest difference weren&#39;t about picking tools. They were about where to draw architectural boundaries.</p>

<h3>1. Keeping templates opinionated was the right call</h3>

<p>Abridge started with a small set and resisted the urge to add templates every time someone had a slightly different use case. The Pareto principle holds: seven well-maintained templates with solid shared modules cover the vast majority of our workloads. The discipline required to say &quot;fit your workflow to an existing template&quot; pays dividends in maintainability.</p>

<h3>2. Separating setup logic from template logic was the other key decision</h3>

<p>Early on, templates were doing too much. Moving repo-specific setup into <code>coder_setup.sh</code> kept templates stable and let teams iterate on their own tooling without waiting for an infra change. It also made the compliance boundary cleaner: templates encode what is and isn&#39;t allowed at the network and compute level, and repos handle everything above that.</p>

<h3>3. The ephemeral vs. persistent data model required more upfront thought than we expected</h3>

<p>Learning that the hard way—discovering that data doesn&#39;t persist after a workspace restart —is a bad experience for researchers in the middle of a training run. We now document it explicitly, reinforce it in template descriptions, and provide shared persistent storage where the workload genuinely needs it.</p>

<p>As Abridge continues to scale, the infrastructure underneath has to scale with it. Fractional GPU allocation, better observability into workspace usage, and clearer template versioning are on the roadmap. The goal remains the same: get infrastructure out of the way so clinical scientists, ML researchers, and product engineers can focus on building the product that matters to our customers.</p>


Abridge is one of the fastest-growing AI companies in healthcare. This is the story of how Abridge rearchitected its developer infrastructure using Coder and transformed developer onboarding time from days to minutes.

How Abridge Scaled Developer Infrastructure from Days to Minutes

Taruj Goyal is an ML/Infrastructure Engineer/Claude Whisperer at Abridge who works on scalable developer platforms, GPU-enabled ML environments, Inference Optim

Taruj Goyal

Explore the latest from Taruj

How Abridge Scaled Developer Infrastructure from Days to Minutes

Related content

Coder Named to Redpoint’s 2026 InfraRed 100

In The Age Of AI, Dev Workloads Are Cloud Workloads

Deploying AI Agents at Scale Without Sacrificing Control & Governance