Spike Curtis

Principal Engineer

When you build developer infrastructure that thousands of engineers depend on, your own development environment has to hold up under pressure. At Coder, we use Coder to build Coder. That means our developers work on large shared machines with significant CPU and memory, and running Go&#39;s parallel test suite to shorten the inner code-test loop. That setup comes with real advantages. It also comes with costs, and one of them quietly compounded for years before we finally dealt with it.

By late 2025, running <code>make test</code> consumed nearly 25GB of memory. Enough to put strain even on our large development machines and slow down every engineer working on the platform. This kind of problem doesn&#39;t announce itself — it just gradually makes everything a little worse until you stop and ask why.

This post walks through how we tracked down the root cause, what we found, and how a two-line fix delivered a 70% reduction in test memory.

<h1>Starting the investigation</h1>

We had an initial hunch that maybe the memory allocations were attributable to Postgres, but it’s important to avoid drawing conclusions too early. We clone a fresh database on a real Postgres server for each of our integration-style test cases. Since our database was containerized in Docker for testing, we could monitor its memory usage on the CLI using <code>docker stats</code>.

<img src="https://www.datocms-assets.com/19109/1772835151-1_code-snippet.png" alt="">

While 1GB is nothing to sneeze at, it was a far cry from the observed 25GB in use looking at system stats. With Postgres exonerated, we turned to the Go tests themselves.

Using <code>top</code> to monitor memory by process:

<img src="https://www.datocms-assets.com/19109/1772835178-2_code-snippet.png" alt="">

We could see different Go modules executing in parallel, some of which, like <code>coderd.test</code> in this example, using multiple GB on their own. Across multiple packages this is where the majority of the memory was being used.

<h1>Profiling the memory</h1>

The next step was to get even more precise. We profiled where the memory is being used by our tests by using Go’s built-in <code>-memprofile</code> flag, like this:

<img src="https://www.datocms-assets.com/19109/1772835208-3_code-snippet.png" alt="">

From there, we used the <code>go tool pprof</code> to inspect the memory profile. The Top view shows function calls ranked by memory usage, and the answer was immediate: roughly 71% of the in-use memory, that is, multiple GB, was for Wireguard message buffers.

To understand why, you need to understand how Coder uses Wireguard both in production and in testing.

<h1>How Wireguard fits into Coder</h1>

We encrypt all packets to and from Coder workspaces using Wireguard. In production instances of the Coder Server, Workspace Agent, Workspace Proxies, and the CLI, we run a single Wireguard device instance in the process, even if we connect to multiple Workspaces as the Server and Workspace Proxies do.

<img src="https://www.datocms-assets.com/19109/1772835248-4_diagram.png" alt="">

In testing, however, the picture looks very different. We have a large number of integration-style tests that create a Coder Server, complete with a Wireguard device. In some of these tests we also create Workspace Agents or CLI instances that create additional Wireguard devices. Thus, each of these tests creates 1-3 Wireguard devices. The largest packages have nearly 1000 integration-style test cases like this.

In our case, it was not any recent change that caused a large spike in memory usage. Automated software testing is a cumulative process, and over the years, like a frog in a pot of water slowly heating up, we didn’t take notice of the ever-increasing peak memory usage.

<h1>Understanding the allocation</h1>

Wireguard encrypts individual IP packets and then encapsulates them in UDP for transmission over the network. In order to read incoming UDP packets on a socket, the owning process allocates memory buffers from its own address space, then calls into the OS kernel to ask it to write the incoming packets to the buffer. This creates overhead in switching from user-mode to kernel-mode execution, and so for high performance this request can be batched. The stack trace we saw in our <code>pprof</code> view is Wireguard allocating these buffers for batched reads from the operating system.

<img src="https://www.datocms-assets.com/19109/1772835285-5_code-snippet.png" alt="">

and

<img src="https://www.datocms-assets.com/19109/1772835335-6_code-snippet.png" alt="">

Wireguard allocates <code>maxBatchSize</code> buffers of size <code>MaxMessageSize</code> ( = 64kB on Linux, the maximum size of a UDP payload). When paired with Tailscale like in Coder, the <code>maxBatchSize</code> is set by the Wireguard constant <code>IdealBatchSize</code> ( = 128). So we have buffer allocations of 128 * 64 kB = 8 MB, or 16 MB on a dual IPv4 / IPv6 system. This is small in the context of per-process memory usage in production, but as we have seen adds up to GB in our tests.

<h1>Lowering the peak</h1>

From here we considered several options.

The reason that Coder Server (and the Workspace Proxy) have a Wireguard device is so that they can proxy HTTP requests to the Workspace when requested by an authorized user. A minority of test cases actually do this, so we could simply disable creating a Wireguard device in tests that don’t need it. But, figuring out which tests need the Wireguard device and which don’t would be an expensive undertaking in our multi-thousand-case suite of tests. Furthermore, it would still leave the memory allocations in the tests that did proxy HTTP requests.

When communication with workspaces is needed, at a high level, the Go code just expects to get a <code>net.Conn</code> instance: the actual Wireguard device underpinning it is very deep and abstracted away. This suggested a second option: mocking out the Workspace networking subsystem (which we call <code>tailnet</code> because it embeds the <a href="https://github.com/tailscale/tailscale/">open source Tailscale</a> software). A lightweight, per-test mock could serve connections using the in-memory <a href="https://pkg.go.dev/net#Pipe">net.Pipe()</a> call. This would drop the memory allocation to essentially nil, but at the cost of a significant refactor, and would reduce the test coverage of the actual networking subsystem.

The option we went with in the end involved a deep dive into Wireguard internals.

It’s straightforward to verify that <code>IdealBatchSize</code> is a purely performance optimization constant. We can change its value and get the same functional behavior for our Wireguard device. The choice of 128 for production systems balances latency and memory usage against the overhead of making more calls to the OS kernel. In our testing scenarios we send very little data over Wireguard. Most tests send nothing, and those that do often exchange a single HTTP request/response.

Our <a href="https://github.com/coder/wireguard-go/pull/5">fix is just a couple lines of code</a>, taking advantage of Go build flags to ensure we only make the change in tests.

<img src="https://www.datocms-assets.com/19109/1772835372-7_code-snippet.png" alt="">

We could go even further by also reducing the <code>MaxMessageSize</code> but it’s a little less straightforward to verify this doesn’t change functional behavior. Buffers smaller than the received UDP packets cause truncation and for Wireguard, truncated means discarded. While the buffers were the single largest contributor, they “only” represented 71% of the memory. <a href="https://en.wikipedia.org/wiki/Amdahl%27s_law">Amdahl’s Law</a> applies here: our 32x reduction in buffer memory (in percentage terms, a 97% reduction) translates to just shy of 69% reduction overall. A further reduction by 32x (that is 1024x overall) only ekes out another 3% reduction (71% overall).

We can use <code>go tool pprof</code> to examine the savings, using the <code>-diff_base</code> flag to compare with previous:

<img src="https://www.datocms-assets.com/19109/1772835871-8_code-snippet.png" alt="">

<h1>Key Takeaways</h1>

Two lines of code. A 69% reduction in peak test memory. The fix itself was straightforward — but only because we resisted the urge to act on our first instinct and instead let the data lead us there.

Postgres seemed like the obvious culprit. It wasn&#39;t. The real cause was buried deep in Wireguard&#39;s allocation behavior, accumulating silently across thousands of test cases over years of growth. Profiling early with <code>-memprofile</code> and <code>go tool pprof</code> made the answer clear in minutes. From there, Go build flags gave us a clean, safe way to apply the fix without touching production behavior at all.

The test suite is faster. The machines breathe easier. And the next time something quietly compounds in the background, we know the playbook.

When you build developer infrastructure that thousands of engineers depend on, your own development environment has to hold up under pressure. At Coder, we use Coder to build Coder. That means our developers work on large shared machines with significant CPU and memory, and running Go's parallel test suite to shorten the inner code-test loop. That setup comes with real advantages. 

How changing one constant reduced our CI memory usage by 70%

How Changing One Constant Reduced Our CI Memory Usage By 70%

Coder’s engineering team has discovered a serious security vulnerability in the Go standard library <code>database/sql</code> package. Attacks based on this vulnerability are application and SQL-driver specific, but could allow a remote attacker to modify SQL query results, with potentially disastrous consequences up to full application compromise.

We highly recommend engineering teams using <code>database/sql</code> update to Go version 1.24.6 immediately.

The Coder application is patched in versions 2.25.1, 2.24.3, and 2.23.5. We highly recommend all Coder deployments upgrade to a patched release.

This vulnerability is assigned <a href="https://github.com/golang/go/issues/74831">CVE-2025-47907</a>.

<h2>Impact and outlook</h2>

When a query running in a vulnerable version of Go is canceled, there exists a data race where the SQL driver’s connection object can be returned to the pool of available connections and reused for a second query while the first query is still decoding the results. If the SQL driver retains any memory, such as internal buffers (fairly common), this memory could be overwritten by the second query, corrupting the results of the first query. It’s similar to the <a href="https://cwe.mitre.org/data/definitions/416.html">“use after free”</a> class of bugs, but here we are returning a shared resource to a resource pool instead of returning raw memory to a memory manager.

database/sql data race

A deliberate attack based on this vulnerability is application specific and SQL Driver specific: it involves knowing the memory layout of internal buffers and application-specific queries and results.

As of writing there are no known exploits against any applications.

We expect that developing a successful exploit is difficult because it is triggered by narrow timing window during which the first query’s <code>context.Context</code> must be canceled while a result row is being decoded (”scanned” in the SQL driver terminology) into Go data types. Then the SQL driver must be reused for a second query all before the first query decode completes. Furthermore, to be a successful exploit, the application needs to use the results of the SQL query even after the <code>context.Context</code> has been canceled.

That’s the good news: that we expect it’s hard to exploit. The bad news is that a successful exploit could be very severe. If an attacker were able to corrupt the results of a SQL query, the impact is limited only by the security implications of that query. Some examples:

<ul>
<li>If a query fetching a cryptographic authentication key were compromised, an attacker could bypass access controls and take complete control of the application.</li>
<li>If a query fetches executable code or scripts, a successful exploit could result in remote code execution.</li>
</ul>

<h2>How we identified the vulnerability</h2>

The vulnerability has been hiding in plain sight in <code>database/sql</code> since at least Go version 1.10. As early as at least 2021, developers have been hitting the issue, but <a href="https://github.com/lib/pq/issues/1021">incorrectly diagnosed it as an issue in the <code>lib/pq</code> driver</a>, rather than in the Standard Library, and seem to have missed the security implications. Our team also initially missed the security implications when we started seeing data races detected by the <a href="https://go.dev/doc/articles/race_detector">Go race detector</a> in our CI.

My spidey sense was tingling when I got involved in looking at <a href="https://github.com/coder/internal/issues/731">the issue in June</a>. The output of the race detector was showing a data race while reading data from a SQL query:

<pre><code>Previous read at 0x00c00120ef3c by goroutine 2187:
 runtime.slicecopy()
 /opt/hostedtoolcache/go/1.24.4/x64/src/runtime/slice.go:355 +0x0
 bytes.Clone()
 /opt/hostedtoolcache/go/1.24.4/x64/src/bytes/bytes.go:1412 +0xfbc
 database/sql.convertAssignRows()
 /opt/hostedtoolcache/go/1.24.4/x64/src/database/sql/convert.go:272 +0xfbd
 database/sql.(*Rows).Scan()
 /opt/hostedtoolcache/go/1.24.4/x64/src/database/sql/sql.go:3400 +0x727
 github.com/coder/coder/v2/coderd/database.(*sqlQuerier).GetAPIKeysLastUsedAfter()
 /home/runner/work/coder/coder/coderd/database/queries.sql.go:315 +0x667
 github.com/coder/coder/v2/coderd/database.(*sqlQuerier).GetAPIKeysLastUsedAfter()
 /home/runner/work/coder/coder/coderd/database/queries.sql.go:307 +0x154
(truncated for brevity)
</code></pre>

That is, while decoding API Keys from the database, some other goroutine was modifying the data we were decoding! The writing go routine we found in that CI run was just closing the database connection. Not something with any possible user-controlled parameters, so I was intrigued but not yet worried.

I sounded the alarm internally when I later discovered a more damning data race in our CI, which showed two separate queries involved as the reading and writing goroutines, respectively. This proved that it was possible for user-controlled data from an unrelated query to leak into the results.

Having never written a Go <code>sql.Driver</code> myself, I also initially assumed the vulnerability was in the <code>lib/pq</code> driver that we use: that it was retaining a reference to a buffer is was supposed to have passed control of to the <code>database/sql</code> package. But, with some more research into the contract and inline comments in the <code>database/sql</code> package, it became clear to me that instead the issue was a narrow race condition in which the <code>*sql.Rows</code> object could be closed by context cancelation while a <code>Scan()</code> call was still in progress. From the perspective of a SQL driver, the buffers used to return results were fair game to re-use since the <code>Rows</code> had been closed.

<h2>Take aways for Go developers</h2>

Our team’s primary role is developing our own applications, not hunting for security issues: we are not security researchers by training. Still, security is everyone’s responsibility and we strive to be vigilant in our engineering practice.

<ul>
<li>Run the Go race detector over as much of your CI test suite as you can. Data races are fertile ground for security vulnerabilities, so give them extra attention when your tests shake them out.</li>
<li>Investigate any integrity issues in the persistence layer. For us, and many applications like us, all authentication and authorization decisions ultimately rest on data we retrieve from a SQL database, so a compromise there is as severe as it gets.</li>
<li>There is no substitute for a full root cause analysis. When we started seeing the data races, we discussed moving to a different PostgreSQL driver. While it might still be a good idea in terms of something that is more actively maintained, if we just switched and moved on we could still be vulnerable today, since it was not the driver that had the issue.</li>
</ul>

We would like to acknowledge and thank the Go team for their attention and help in getting this issue resolved.

Coder’s engineering team has discovered a serious security vulnerability in the Go standard libary `database/sql` package. As of writing there are no known exploits against any applications, but we recommend updating to Go version 1.24.6 immediately and upgrading all Coder deployments to a patched release.

Query-Mutating Data Race in Go: Hiding in Plain Sight

I hate flaky tests.

We repeatedly run our suite of tests during development time and on every commit, so if a test is flaky, developers quickly learn not to trust it. It’s so easy to just retry and assume everything is fine. This is the worst of both worlds because the test isn’t providing value as an indication that something needs to be fixed and it drags out the review-merge cycle with test reruns. 

Across a modest-sized development team, even a test that flakes 1 in 100 will burn one of your devs every day on average.

Today we’re going to dive deep into one particular area that’s hard to get right: code that depends on time.

<h2>Repeatability and Determinism</h2>

If your code is supposed to do something at 8pm every day, you can’t just wait around until 8pm to test that it does the right thing. Similarly, if your code is supposed to do something after 30 minutes, you probably don’t want to wait 30 minutes to know if it’s working correctly.

One approach is to make the values “8pm” and “30 minutes” configurable, and choose them at test time. This can work, but it has drawbacks.

One is repeatability: the tests will be operating on different times each run, and on different times than they would in production. I’ve seen flakes related to daylight savings time and whether the test happens to start at zero minutes past the hour.

The second is determinism. Compress 30 minutes into 30ms and your tests may no longer do things in the order you expect, since consumer operating systems don’t make hard guarantees about real-time processing. Writing tests this way is an explicit balancing act between fast and flake-free execution. It’s hard to get right, especially since development environments tend to have fast hardware and not a lot of resource contention, whereas CI runs on constrained and shared hardware.

<h2>Introducing Quartz</h2>

Today we are <a href="https://github.com/coder/quartz" target="_blank">introducing Quartz</a>, a mocking library in Go for testing code that depends on time.

By mocking out calls that query or depend on the real time, we can write unit tests that are repeatable, deterministic, and execute as quickly as the CPU allows. This is, in itself, not a new idea. In building Quartz, we took some inspiration from:

<ul>
<li><a href="http://github.com/benbjohnson/clock" target="_blank">github.com/benbjohnson/clock</a></li>
<li><a href="https://github.com/coder/tailscale/blob/main/tstest/clock.go" target="_blank">Tailscale&#39;s tstest.Clock</a></li>
<li><a href="http://github.com/aspenmesh/tock" target="_blank">github.com/aspenmesh/tock</a></li>
</ul>

Quartz shares the high level design of a Clock interface that closely resembles the functions in the time standard library, and a &quot;real&quot; clock passes through to the standard library in production, while a mock clock gives precise control in testing.

For several reasons, it is a tall order to write tests that are repeatable, deterministic and fast, when it comes to code that depends on time. The rest of this blog post will discuss what these reasons are and why we found these existing libraries insufficient. If you are just ready to dive into unit testing with Quartz, head over to the repository now and look at the <a href="https://github.com/coder/quartz?tab=readme-ov-file#readme" target="_blank">README</a> and <a href="https://github.com/coder/quartz/blob/main/example_test.go" target="_blank">examples</a>.

I also want to give a big thank you to the authors of the above libraries. Discussing the limitations of the libraries in the next sections is not meant to imply these libraries are bad or not useful. To the extent that Quartz surpasses these limitations, it is only by standing on their shoulders.

<h2>Preventing Test Flakes</h2>

The following example comes from the README from benbjohnson/clock:

<pre><code>mock := clock.NewMock()
count := 0

// Kick off a timer to increment every 1 mock second.
go func() {
 ticker := mock.Ticker(1 * time.Second)
 for {
 &lt;-ticker.C
 count++
 }
}()
runtime.Gosched()

// Move the clock forward 10 seconds.
mock.Add(10 * time.Second)

// This prints 10.
fmt.Println(count)
</code></pre>

The first race condition is fairly obvious: moving the clock forward 10 seconds may generate 10 ticks on the <code>ticker.C</code> channel, but there is no guarantee that <code>count++</code> executes before <code>fmt.Println(count)</code>.

The second race condition is more subtle, but <code>runtime.Gosched()</code> is the tell. Since the ticker is started on a separate goroutine, there is no guarantee that <code>mock.Ticker()</code> executes before <code>mock.Add()</code>. <code>runtime.Gosched()</code> is an attempt to get this to happen, but it makes no hard promises. On a busy system, especially when running tests in parallel, this can flake, advance the time 10 seconds first, then start the ticker and never generate a tick.

Let&#39;s talk about how Quartz tackles these problems.

In our experience, an extremely common use case is creating a ticker then doing a 2-arm <code>select</code> with ticks in one and context expiring in another, i.e.

<pre><code>t := time.NewTicker(duration)
for {
 select {
 case &lt;-ctx.Done():
 return ctx.Err()
 case &lt;-t.C:
 err := do()
 if err != nil {
 return err
 }
 }
}
</code></pre>

In Quartz, we refactor this to be more compact and testing friendly:

<pre><code>t := clock.TickerFunc(ctx, duration, do)
return t.Wait()
</code></pre>

This affords the mock <code>Clock</code> the ability to explicitly know when processing of a tick is finished because it&#39;s wrapped in the function passed to <code>TickerFunc</code> (<code>do()</code> in this example).

In Quartz, when you advance the clock, you are returned an object you can <code>Wait()</code> on to ensure all ticks and timers triggered are finished. This solves the first race condition in the example.

(As an aside, we still support a traditional standard library-style <code>Ticker</code>. You may find it useful if you want to keep your code as close as possible to the standard library, or if you need to use the channel in a larger <code>select</code> block. In that case, you&#39;ll have to find some other mechanism to sync tick processing to your test code.)

To prevent race conditions related to the starting of the ticker, Quartz allows you to set &quot;traps&quot; for calls that access the clock.

<pre><code>func TestTicker(t *testing.T) {
 mClock := quartz.NewMock(t)
 trap := mClock.Trap().TickerFunc()
 defer trap.Close() // stop trapping at end
 go runMyTicker(mClock) // async calls TickerFunc()
 call := trap.Wait(context.Background()) // waits for a call and blocks its return
 call.Release() // allow the TickerFunc() call to return
 // optionally check the duration using call.Duration
 // Move the clock forward 1 tick
 mClock.Advance(time.Second).MustWait(context.Background())
 // assert results of the tick
}
</code></pre>

Trapping and then releasing the call to <code>TickerFunc()</code> ensures the ticker is started at a deterministic time, so our calls to <code>Advance()</code> will have a predictable effect.

Take a look at <code>TestExampleTickerFunc</code> in <code>example_test.go</code> for a complete worked example.

<h2>Complex Time Dependence</h2>

Another difficult issue to handle when unit testing is when some code under test makes multiple calls that depend on the time, and you want to simulate some time passing between them.

A very basic example is measuring how long something took:

<pre><code>var measurement time.Duration
go func(clock quartz.Clock) {
 start := clock.Now()
 doSomething()
 measurement = clock.Since(start)
}(mClock)

// how to get measurement to be, say, 5 seconds?
</code></pre>

The two calls into the clock happen asynchronously, so we need to be able to advance the clock after the first call to <code>Now()</code> but before the call to <code>Since()</code>. Doing this with the libraries we mentioned above means that you have to be able to mock out or otherwise block the completion of <code>doSomething()</code>.

But, with the trap functionality we mentioned in the previous section, you can deterministically control the time each call sees.

<pre><code>trap := mClock.Trap().Since()
var measurement time.Duration
go func(clock quartz.Clock) {
 start := clock.Now()
 doSomething()
 measurement = clock.Since(start)
}(mClock)

c := trap.Wait(ctx)
mClock.Advance(5*time.Second)
c.Release()
</code></pre>

We wait until we trap the <code>clock.Since()</code> call, which implies that <code>clock.Now()</code> has completed, then advance the mock clock 5 seconds. Finally, we release the <code>clock.Since()</code> call. Any changes to the clock that happen before we release the call will be included in the time used for the <code>clock.Since()</code> call.

As a more involved example, consider an inactivity timeout: we want something to happen if there is no activity recorded for some period, say 10 minutes in the following example:

<pre><code>func (i *InactivityTimer) Start() {
 i.mu.Lock()
 defer i.mu.Unlock()
 next := i.clock.Until(i.activity.Add(10*time.Minute))
 t := i.clock.AfterFunc(next, func() {
 i.mu.Lock()
 defer i.mu.Unlock()
 next := i.clock.Until(i.activity.Add(10*time.Minute), &quot;inner&quot;)
 if next == 0 {
 i.timeoutLocked()
 return
 }
 t.Reset(next)
 })
}
</code></pre>

All Quartz <code>Clock</code> functions, and functions on returned timers and tickers support zero or more string tags that allow traps to match on them.

<pre><code>func TestInactivityTimer_Late(t *testing.T) {
 // set a timeout on the test itself, so that if Wait functions get blocked, we don&#39;t have to
 // wait for the default test timeout of 10 minutes.
 ctx, cancel := context.WithTimeout(10*time.Second)
 defer cancel()
 mClock := quartz.NewMock(t)
 trap := mClock.Trap.Until(&quot;inner&quot;)
 defer trap.Close()

 it := &amp;InactivityTimer{
 activity: mClock.Now(),
 clock: mClock,
 }
 it.Start()

 // Trigger the AfterFunc
 w := mClock.Advance(10*time.Minute)
 c := trap.Wait(ctx)
 // Advance the clock a few ms to simulate a busy system
 mClock.Advance(3*time.Millisecond)
 c.Release() // Until() returns
 w.MustWait(ctx) // Wait for the AfterFunc to wrap up

 // Assert that the timeoutLocked() function was called
}
</code></pre>

This test case will fail with our bugged implementation, since the triggered AfterFunc won&#39;t call <code>timeoutLocked()</code> and instead will reset the timer with a negative number. The fix is easy, use <code>next &lt;= 0</code> as the comparison.

<h2>Correspondence Principle</h2>

One final criterion in designing Quartz’s API was the desire to be able to write unit tests that are easy to understand. 

Time flows in one direction, from past to future, and I wanted unit tests written with Quartz to have the same monotonic flow and definite ordering. As you read a unit test from top to bottom, time advances forward, and you can define explicit breaks in the flow to inspect and assert the state of code under test, like breakpoints in debugger, or hitting the pause button while playing back a video.

Consider this example unit test from the Quartz repository. The color coding shows how time advances as the unit test executes.

<img src="https://www.datocms-assets.com/19109/1720191961-code-example.png" style="width:100%; border: 1px solid #262626; border-radius:5px;"/>

From the perspective of the unit test function, the clock is always in a well defined state, and only changes state explicitly and synchronously.

The second ingredient for writing repeatable, deterministic unit tests is for the unit under test to be in a well defined state, with no races, when you make your test assertions. Quartz cannot guarantee this on its own, but it is designed to make this as easy as possible via

<ol>
<li>Waiting for <code>Advance/AdvanceNext</code> calls to complete. When paired with <code>AfterFunc</code> and <code>TickerFunc</code>, this allows you to wait for the unit under test to complete its timeout or tick processing. You can then assert the results of that processing without any races.</li>
<li>Trapping calls into the Quartz library via the Clock interface, Timers and/or Tickers. These allow you to suspend execution of the unit under test at deterministic points, where you can make assertions or advance the clock.</li>
</ol>

These two properties–clock in a well defined state, unit under test in a well defined state–mean that you can make test assertions without worry that they will flake. Polling (e.g. “Eventually”), sleeping, or calling runtime.Gosched() within your tests are all indications that your tests are not fully deterministic. It’s still possible to write tests that don’t flake this way, but it’s harder and you make trade-offs with speed.

<h2>Present and Future of Quartz</h2>

The current release is v0.1.0, reflecting the fact that we are not yet making an ironclad promise not to change the API before v1.0. I don’t anticipate any changes, but it seems premature to make that promise before Quartz sees much deployment outside of Coder.

Having used Quartz internally at Coder for some time, we feel it is stable enough for general, public use. Just don’t robotically upgrade the library without checking until we declare the v1.0 API, likely later this year.

<a href="https://github.com/coder/quartz" target="_blank">Please give Quartz a try</a>. If you like it, star it! If you have any issues or feature requests, please write a GitHub issue on the repo.



1 You may retort that testing at different times of day and seasons of the year gives you more test coverage. For what it’s worth, both these flakes were test bugs related to computing the expected outputs, not product bugs. To the extent your code is sensitive to these factors, you want test cases for them, not the luck of the draw of when the test is run. That is, you don’t want to wait around until daylight savings time to know whether your code will work when it does.

Quartz shares the high level design of a Clock interface that closely resembles the functions in the time standard library, and a "real" clock passes through to the standard library in production, while a mock clock gives precise control in testing.

Introducing Quartz: A Deterministic Time Testing Library for Go

At Coder we’re in the business of making cloud development environments awesome. When you are developing remotely, the speed and quality of the connection you have to your workspace is a big driver of the overall experience you’ll have. One important measure of how good your connection is:

Throughput - the rate at which you can send or receive data across a network

Throughput is usually quoted in bits per second, and on modern internet networks, these range from a few millions of bits per second (Mb/s) to a few billions of bits per second (Gb/s).

You will really notice throughput differences when copying files to and from your workspace, or testing web applications that push heavy assets.

Coder v2.12, which we have just released, includes several networking improvements that together increase throughput dramatically: over 5x in some cases. Below is a comparison of throughput test results for 3 different versions of Coder, across 3 different AWS environments with different latencies. They are all based on t3.xlarge VM instances. Your throughput will vary based on many factors besides latency including the speed of your connection and network congestion.

<img src="https://www.datocms-assets.com/19109/1717161538-frame-1.png" style="width:100%; border: 1px solid #262626; border-radius:5px;"/>

The 60ms test is from us-east-1 to us-west-1, and is a typical latency you might experience connecting to a Coder Workspace relatively far from home, say, across the continental USA.

The 10ms test is from us-east-1 to us-east-2, and is a typical latency you might experience if you are connecting to a Coder Workspace within the same geographic region.

The 1ms test is within us-east-1, and is an extremely low latency connection you might only achieve if you are connecting to a Coder Workspace hosted by your company on-premise, within the same building or corporate campus.

The rest of this post is a deep-dive into how we achieved these throughput increases. Coder builds an end-to-end encrypted tunnel to your workspace, and as such the theoretical maximum throughput to your workspace will be slightly lower than that of the underlying network path due to tunnel overheads, but in our testing we found a significant contribution to reduced throughput was the performance of the embedded TCP stack.

<h2>Wait, Coder Has an Embedded TCP Stack?</h2>

The TCP stack is not something that most software engineers think about that often, even those that are working deep in the backend. Most application software, even server software, does not have a TCP stack embedded even though TCP is essential to the web and the majority of application protocols. The reason is simple: it’s handled in the OS! When you want to speak TCP you just ask the OS to open a socket for you.

So, why doesn’t Coder do that?

We are committed to keeping your data safe through end-to-end encryption and to making Coder easy to run across a wide variety of systems from client laptops and desktops to VMs, containers, and bare metal. If we used the TCP implementation in the OS, we’d need a way for the TCP packets to get from the operating system back into Coder for encryption. This is called a TUN device in unix-style operating systems and creating one requires elevated permissions, limiting who can run Coder and where. Asking for elevated permissions inside secure clusters at regulated financial enterprises or top secret government networks is at best a big delay and at worst a nonstarter.

Instead, Coder embeds the networking stack from gVisor. gVisor is a container runtime that reimplements the entire Linux ABI (syscalls) in Go, but we only need the networking for our purposes. gVisor is deployed extensively by Google in their Cloud offerings, so it is mature and stable, but in our testing, we find that it is not always optimized for our use case.

Google uses gVisor within the data center, where networks have high bandwidth and extremely low latency (hundreds of μs to few ms). Coder users have 10x to 100x more latency, and this can have a big difference in network throughput. We’ve had to make some changes both to how gVisor is configured and to gVisor itself to make it perform well for Coder users.

<h2>A Tale of TCP Buffers</h2>

The promise that a TCP socket makes is that it will deliver a stream of bytes for you, and it will handle reordered or dropped packets, so the receiving application gets your data in order.

TCP uses buffers both when sending and receiving data to handle this. When sending data, it keeps a copy of the data in the buffer until the other TCP stack explicitly acknowledges receipt, in case it needs to be retransmitted. When receiving data, it keeps out-of-order data in the buffer until the in-order data arrives.

TCP data is being tracked in the buffers, and can’t be deleted until it’s delivered in order to the receiving application, which takes some time (the latency). Because of this, the maximum theoretical throughput you can achieve with TCP is directly proportional to the buffer sizes, and inversely proportional to the latency.

<img src="https://www.datocms-assets.com/19109/1717105039-throughput.png" width="128"/>

Where T is the throughput, C is some constant (approximately 1), B is the buffer size 1, and L is the latency.

This might be a surprising result at first! We often explain packet switched networking with the analogy of the postal system. If I live in California and am writing letters to my friends in Texas and Florida, it’s easy to imagine that I could post as many letters as I want to them. Sure, the letters might take an extra day to reach my friend in Florida, but the rate at which I send them and the rate at which they could arrive shouldn’t be fundamentally limited by that delay.

The key here is that we’re ensuring reliable delivery. The postal system, in our little analogy, loses letters all the time and doesn’t even offer registered mail2. So, I have to keep copies of all my letters until my friends write back to say they got them. If I send 10 letters per day, and it takes 2 days to deliver to Texas (and 2 more days to get a reply), then I better have space for 40 letters. If I try to do the same for Florida, which takes 3 days each way, then I better have space for 60 letters, or risk losing some of the letters permanently.

Let’s see what this looks like in TCP. The Coder CLI includes a speedtest command to test throughput to your workspace, and can capture the TCP packets for analysis in Wireshark. (You will need the Tailscale Wireshark Lua plugin to decode the packet captures.) The following runs a 20 second upload test to my workspace named <code>dogfood2</code>.

<pre><code>coder speedtest --direction up --pcap-file dogfood2-up-00.pcap -t 10s dogfood2
</code></pre>

Testing with Coder 2.9, I get something like:

<pre><code>Starting a 10s upload test...
INTERVAL THROUGHPUT 
0.00-1.28 sec 65.2983 Mbits/sec 
1.28-2.58 sec 51.8875 Mbits/sec 
2.58-3.86 sec 52.3992 Mbits/sec 
3.86-5.09 sec 54.5126 Mbits/sec 
5.09-6.31 sec 54.8532 Mbits/sec 
6.31-7.55 sec 54.4550 Mbits/sec 
7.55-8.77 sec 54.9556 Mbits/sec 
8.77-10.00 sec 54.3970 Mbits/sec 
-----------------------------------
0.00-10.00 sec 55.3624 Mbits/sec 
</code></pre>

I opened one of the packet capture files and asked Wireshark to plot the TCP window (green line) and number of unacknowledged bytes in flight (blue line).

<img src="https://www.datocms-assets.com/19109/1717155989-tcp1.png" style="width:100%"/>

The TCP window is, essentially, how much room the receiver has in its buffer, and the TCP sender is not allowed to send more unacknowledged data than this. When the receiver acknowledges some data, the blue line dips down and we send more. This shows us that the TCP throughput is being limited by the buffer sizes, not the underlying network bandwidth.

The TCP window hovers right around 512 KB, which is one half the default TCP receive buffer size in gVisor. The maximum TCP window is capped at half the buffer to allow for out of order segments and for data not yet delivered to the application.

Increasing the buffers gives the TCP stack the room it needs to track more data in flight, increasing throughput (dramatically in some cases).

On 2.10, which has increased TCP buffers, here is an example speedtest:

<pre><code>Starting a 10s upload test...
INTERVAL THROUGHPUT 
0.00-1.50 sec 66.9476 Mbits/sec 
1.50-2.55 sec 191.7347 Mbits/sec 
2.55-3.58 sec 245.4543 Mbits/sec 
3.58-4.74 sec 129.6934 Mbits/sec 
4.74-5.74 sec 184.2218 Mbits/sec 
5.74-6.75 sec 200.6757 Mbits/sec 
6.75-7.82 sec 204.1865 Mbits/sec 
7.82-8.83 sec 215.8954 Mbits/sec 
8.83-9.89 sec 204.8810 Mbits/sec 
9.89-10.06 sec 203.5317 Mbits/sec 
------------------------------------
0.00-10.06 sec 176.8486 Mbits/sec 
</code></pre>

<h3>When Slow Start is Too Fast</h3>

In my performance testing, after increasing buffer sizes I saw much improved throughput overall, but I noticed that my connection speed would often stutter in the first couple seconds.

Here is a Wireshark plot of the TCP throughput for my connection.

<img src="https://www.datocms-assets.com/19109/1717156014-tcp2.png" style="width:100%"/>

It shows the throughput ramping up in the first second or so, then coming to a screeching halt around 2 seconds, before recovering and stabilizing. This happens even though we never get close to saturating the TCP window we discussed in the last section.

While you’d hardly notice the issue downloading a 100MB file, this stutter would turn a 2 second page load into 3-4 seconds.

To understand what’s happening we need to dive a bit deeper into TCP and talk about congestion control. TCP congestion control algorithms moderate the amount of data a TCP connection sends to avoid overwhelming congested network links along the path. Avoiding this is both in our own selfish interest, since retransmitting dropped packets lowers our effective throughput, and in the common interest of the Internet as a whole because it allows different applications to share the network capacity.

TCP congestion control works by maintaining a second window variable in addition to the one updated by the receiver. The congestion window is an internal state variable that doesn’t appear in the TCP protocol at all. TCP stack takes the smaller of the send window and congestion window to determine an effective window that limits the amount of data it can send.

The full details of congestion control algorithms are beyond the scope of this blog post, but the central idea is that they detect congestion in the network and then update the congestion window, cutting its size in the face of congestion and slowly opening up the window when it seems the network can handle more.

<img src="https://www.datocms-assets.com/19109/1717156008-tcp3.png" style="width:100%"/>

When network links receive more data than they can carry, routers in the path respond to this congestion by dropping packets. TCP detects these packet drops through missed acknowledgements in the protocol. It resends the data, but also reduces the congestion window in response.

At the very start of a TCP connection, the sending stack doesn’t know anything about how much available bandwidth there is on the network path to the destination, so it starts the congestion window very small: just one packet’s worth of data. It does this to avoid dumping a ton of data into the network all at once and causing a cascade of dropped packets. This is called TCP Slow Start.

However, it does not stay slow for long! During TCP Slow Start, each acknowledged packet of data also increases the congestion window, causing 2 packets to be sent in response. This is an exponential increase in throughput–a doubling every round-trip time (RTT). You can see this in the Wireshark throughput plot above during the first second. When TCP detects congestion, it exits Slow Start, having dynamically discovered the current available bandwidth of the network path.

This exponential increase can often be too aggressive. If we depend on a dropped packet to detect congestion, it can take up to one RTT before we receive an indication. During that time, we double the amount of packets we send out into the network, and we might overshoot by a large amount. This is especially a problem for connections with large bandwidth and high latency, as this combination means a large number of packets are “in flight” in the network at any given time. Overshooting can cause a significant fraction of these in flight packets to be dropped. Multiple dropped packets cause a lot of problems for the TCP stack, including slamming the congestion window shut and increased CPU utilization as the stack has to compute which packets to resend. This is the stuttering I saw in my TCP connections.

Fortunately, I’m far from the first person to have noticed this problem. Researchers Ha &amp; Rhee described the issue and an algorithmic solution in their 2011 paper <a href="https://dl.acm.org/doi/10.1016/j.comnet.2011.01.014" target="_blank">Taming the elephants: New TCP slow start</a>. Their proposed algorithm, called HyStart (short for hybrid start), was eventually implemented in the Linux kernel and a slightly modified version (called HyStart++) implemented in Windows and <a href="https://datatracker.ietf.org/doc/rfc9406/" target="_blank">described in an RFC</a>. HyStart works by tracking slight variations in the round-trip-time to detect network congestion before packets are dropped.

gVisor, the TCP stack we use in Coder, had not implemented HyStart. Google uses gVisor in their data centers, and with extremely short RTTs between nodes, they hadn’t seen the stuttering TCP connections. But, they were very happy to accept a PR when I reached out, and so we implemented and <a href="https://github.com/google/gvisor/pull/10287" target="_blank">upstreamed HyStart to gVisor</a> earlier this spring. We are pleased to include an updated gVisor TCP stack with HyStart starting in Coder 2.12.

Here is a Wireshark plot of TCP throughput with HyStart implemented:

<img src="https://www.datocms-assets.com/19109/1717156000-tcp4.png" style="width:100%"/>

<h3>Packets Lost Within</h3>

Networking is a multi-step process with interacting components, many distributed across different locations, and the nature of such systems is that performance is generally limited by a single, slowest rate limiting step, or bottleneck. Furthermore, when you deal with the problem of a rate limiting step, by making it faster or working around it, the performance rapidly improves up to the limits of what was the second-slowest step.

And so, after dealing with TCP buffers and TCP slow start, we continued to examine Coder’s networking performance. We found that TCP throughput was limited by dropped packets, and that some of the packets were being dropped inside the Coder process before they were sent out on the wire!

Coder uses a modified version of Tailscale for networking. The packet-processing pipeline in this arrangement looks, schematically, like this:

<img src="https://www.datocms-assets.com/19109/1717155994-tcp5.png" style="width:100%"/>

Via some hacked logging, we determined that packets were being dropped in the Channel Endpoint – a software component that looks like a network interface to gVisor and allows WireGuard to read and write network packets like a TUN device.

The Channel Endpoint had an internal queue of packets and when the queue filled up, it dropped packets sent by gVisor’s TCP/IP stack. The Channel Endpoint was a component of gVisor, designed to implement a virtual network link, and dropping packets is what real network links do when they are filled up.

In some sense, this was a reasonable arrangement. If the gVisor stack can send data faster than WireGuard can encrypt, then something needs to put pressure on the stack to get it to back off, and as we discussed in the last section this is exactly what TCP congestion control is designed to do when it detects dropped packets.

However, this is all happening with the same process on a single node, not in some distributed system of interacting routers and switches that make up the Internet. In this context, dropping packets is a very heavyweight solution to the problem of putting back pressure on the TCP stack. Firstly, you have to wait for a network round trip to find out about dropped packets. Secondly, it increases work because you have to resend.

We can do much better, and in fact, Go already includes the data structure we need as a primitive: the buffered channel. A buffered Go channel allows writes to complete without blocking until the buffer is full, and then any further writes simply block until there is room in the channel. This blocks the gVisor stack from sending more data and gives WireGuard time to catch up. And, when WireGuard has caught up, gVisor can resume sending within microseconds, not milliseconds of network round trips. This simple fix doubles throughput on some connections in our test systems.

<h3>Final Words</h3>

I want to end this post by saying that although we have made big strides in the last few Coder versions, we don’t consider this network performance work done. While security of your data in transit is our top priority, we will continue to push for higher performance until we can say confidently that the bottleneck is the underlying network itself.



1 I’m glossing over the difference between the send buffer and the receive buffer here, and in the case where one is significantly smaller than the other, the maximum throughput is limited by the smaller one.

2 This is by design and is called the end-to-end principle. The services offered by the system are kept basic and lightweight, and if you want reliable delivery, or proof of delivery you build it into the protocols the endpoints speak.

Learn how we enhanced TCP performance in Coder to achieve 5X faster throughput by optimizing buffer sizes, implementing the HyStart algorithm, and minimizing packet loss, significantly improving remote development experiences.

How We Improved the Performance of a Userspace TCP Stack in Go by 5X

We develop Coder on Coder which allows our engineers to do their work on big, shared infrastructure (32 cores and 256 GB of RAM in our case). With a lot of hardware, extensive use of t.Parallel() in our Go backend shortens test times considerably.

We also make extensive use of context.Context in our production code, so it appears in our testing. And even if the code under test doesn’t accept a context, it is a very convenient pattern for testing concurrent code.

But, there is a dangerous little sharp edge when using both t.Parallel() and context.WithTimeout(). Tl;dr is–

<blockquote>
Always call t.Parallel() before context.WithTimeout()
</blockquote>

To understand why this is, and what kinds of gnarly test bugs you’ll find by not taking my advice, we need to dig a little bit into what t.Parallel() does under the hood. But first, it’s worth some background on why we make extensive use of context.WithTimeout() in our tests at Coder.

<h1>Test Context Timeouts</h1>

Building programs that correctly handle concurrency is hard, even with Go. With concurrency comes the possibility of race conditions and deadlocks. Here we’re focused on the latter. Consider the following unit test of a hypothetical component that takes inputs and output via channels:

When the component is working fine, the test passes. But if the component is broken and deadlocks, the test also deadlocks. The default timeout on go test is 10 minutes. During the initial development and any maintenance of the component, we’d expect a developer to run the tests iteratively: code, test, code, test…, and so on. If you have to wait 10 minutes to discover a broken test it kills productivity!

You can, of course, change the go test timeout, but that’s a relatively crude instrument as it sets the timeout for the entire test suite, which often grows over time, and different tests might be expected to take different times. I expect most unit tests to execute in less than 1ms, but a complex integration test could take several seconds.

Let’s set a timeout on our test case with a context.

Now, if the component deadlocks, it fails in 100ms. You’ll have to choose this number based on what the unit test is doing, choosing something that is a reasonable multiple of the usual runtime to account for testing on different systems and under load, but still quick enough to avoid dragging out the code/test iterative cycle.

You can make this pattern a lot more readable by adding some helper functions.

<h1>Parallel Tests with Timed Contexts</h1>

Godoc for t.Parallel() is very simple. Perhaps, deceptively simple:

<blockquote>
Parallel signals that this test is to be run in parallel with (and only with) other parallel tests. When a test is run multiple times due to use of -test.count or -test.cpu, multiple instances of a single test never run in parallel with each other.
</blockquote>

The mental model of parallel testing you get from this description looks like this

You might be tempted to think of t.Parallel() as basically a little flag (or like a decorator in other programming languages). Its presence signals something, but you can just kinda throw it anywhere in your test setup code.

If you do this, then your tests might start to mysteriously start to fail with timeouts, even if you haven’t changed the tests or the component you are testing!

In fact, what happens is that t.Parallel() sort of chops your test code into two halves. Any statements before t.Parallel() run serially along with other serial test cases, and then any statements after t.Parallel() run in parallel. Here is a great blog post that covers this in more detail, including how t.Run() is handled.

To keep the diagram readable, I’m going to abbreviate the serial portion of Parallel Test1 (the part before t.Parallel() is called) sPT1, and so on.

Consequently, if you create a timeout context in the serial portion of a parallel test, that is, before the call to t.Parallel(), then it may expire before or during the parallel portion.

Another way to think about this is to just look at the green boxes in the diagram.

A call to t.Parallel() is like a time.Sleep() for an indeterminate time. The time depends on how big the test suite is and how much of it is parallel. So, doing any sort of time-based computation on different sides of the t.Parallel() call is fraught with danger.

Generally, the best thing to do is just put t.Parallel() as the first statement of your test case.

In rare cases, you might need to do some test setup serially, and then want to execute your test cases in parallel. This is often not worth the trouble unless the parallel parts are numerous or take a long time (relative to the setup). But, if you really, really decide you need it, create a parent context without a timeout, tied to the lifetime of the test case (defer testCancel() below). Then create child contexts for before and after t.Parallel() as needed. As the child contexts are tied to the parent’s lifetime, they are also canceled when the test completes.

<h1>Conclusion</h1>

We make extensive use of t.Parallel() at Coder, to the extent our linters yell if you don’t use it. We care a lot about keeping developers in flow here, including our own. I hope I’ve convinced you of the value of test timeouts to avoid disrupting flow during code/test iterations. When you use context timeouts and t.Parallel() in the same test, save yourself some grief and make sure t.Parallel() is first!

This blog explains a common testing bug when using *t.Parallel()* and test timeouts, e.g. *context.WithTimeout()*, and explains how to avoid it.

Go Testing: Contexts and t.Parallel()

I love a good leak hunt. It’s an intellectual mystery, and I always learn something new in the process.

This mystery is like you read about or watch on TV: details are salacious, never depressing, and at the end, we always get a nice logical explanation. That’s the great thing about computers: they are ultimately predictable, so I get to play detective secure in the knowledge that I always get my perp.

If you hate mysteries or are coming back for reference, you can <a href="#key-takeaways">skip to the key takeaways (spoiler warning, duh)</a>.

<h1>Leaky Dog Food</h1>

At Coder, we build tools for managing development workspaces in public and private clouds, and of course, we develop Coder on Coder. Last week, an engineer noticed something suspicious in our dogfood<a href="#1">[1]</a> cluster’s main coder service, <code>coderd</code> (see above).

The restarts are expected because we re-deploy very aggressively to keep close to the tip of master, but you can see each time we restart we begin a slow but steady climb.

Notice the slope of the climb is constant: it increases at the same rate, day or night. This service is our main API server, and its load is definitely not constant. Like most services, it sees the most load during the workday, and nearly nothing overnight. So, our leak is most likely in some constant background task, rather than an API handler.

<h1>Finding Facts</h1>

Ok, so let’s learn some more about our leak!

Fortunately, we run our dogfood cluster with golang’s <a href="https://pkg.go.dev/net/http/pprof">pprof HTTP endpoint</a>. And, we know from metrics we’ve got a goroutine leak, so we can just ask for the stack traces of running goroutines.

At the top of the list, 822 goroutines in a writeLoop and 820 in a readLoop.

<pre><code class="go">goroutine profile: total 2020
822 @ 0x5ac2b6 0x5bbfd2 0x94e4b5 0x5dce21
# 0x94e4b4 net/http.(*persistConn).writeLoop+0xf4 /usr/local/go/src/net/http/transport.go:2392

820 @ 0x5ac2b6 0x5bbfd2 0x94d3c5 0x5dce21
# 0x94d3c4 net/http.(*persistConn).readLoop+0xda4 /usr/local/go/src/net/http/transport.go:2213
</code></pre>

Immediately, we can read off that HTTP is involved. It is a persistConn, which, with <a href="https://github.com/golang/go/blob/8ed0e51b5e5cc50985444f39dc56c55e4fa3bcf9/src/net/http/transport.go#L98">a little bit of code reading</a>, we can see is used in connection pooling on the client side.

Performance HTTP clients, instead of making a new connection for each HTTP request, maintain a pool of connections to each host. These connections are reused for multiple requests before eventually timing out if idle. This increases request &amp; response throughput for issuing many requests to a host, which is very typical of HTTP client applications.

In our code, we appear to be leaking these connections. But how?

Let’s look at <a href="https://github.com/golang/go/blob/8ed0e51b5e5cc50985444f39dc56c55e4fa3bcf9/src/net/http/transport.go#L2213">the line we are hanging at on the readLoop goroutines</a>. It’s a select with a nice, helpful comment above it.

<pre><code class="go">// Before looping back to the top of this function and peeking on
// the bufio.Reader, wait for the caller goroutine to finish
// reading the response body. (or for cancellation or death)
select {
</code></pre>

So, some code is making an HTTP request, then failing to either cancel the request or finish reading the response body. Usually, this happens when the caller doesn’t call <a href="https://pkg.go.dev/net/http#Response"><code>response.Body.Close()</code></a>. This also explains the writeLoop goroutines, which are created as a pair to the readLoop and <a href="https://github.com/golang/go/blob/8ed0e51b5e5cc50985444f39dc56c55e4fa3bcf9/src/net/http/transport.go#L2392">are all waiting for a new HTTP request</a>.

<h1>Needles in Callstacks</h1>

You’d be forgiven for expecting that we are done here. We’ve noticed there is a problem with our metrics, and then used the profiler to track down the exact lines of code that are hanging. Haven’t we solved the mystery?

Well, no. The goroutines we’ve found are working as designed: these guys are the victims, not the perpetrators. The problem is that some other goroutine has made an HTTP request and failed to cancel it or close the response body. And we know the other goroutine(s) are not sitting around waiting for a response, because otherwise we’d see 820 of them in the pprof output. They’ve moved on.

Wouldn’t it be nice to be able to see the call stack where the leaked goroutine was created? You get this kind of output in the <a href="https://go.dev/blog/race-detector">go race detector</a>, so it must be technically possible. Clearly it would add some memory overhead to store it on goroutine creation, but it would likely be a few kB per goroutine. Maybe someday we’ll get a runtime mode that includes it, but unfortunately it would not help here. It’s actually easy to track down <a href="https://github.com/golang/go/blob/8ed0e51b5e5cc50985444f39dc56c55e4fa3bcf9/src/net/http/transport.go#L1750">the line that starts the leaked goroutines</a>, since it’s in the same file. But, remember, this is a persistent connection and it is itself started asynchronously from the HTTP requests that it processes.

No, we need to track down the misbehaving HTTP client code. We know the basic signature of the bug: issue an HTTP request, receive the HTTP response, forget call response.Body.Close(). Our task is gigantic because, simply put HTTP is everywhere. It is the go-to choice for application development protocols. Auditing every HTTP request/response in our codebase would take weeks.

<h1>All in the Timing</h1>

Recall that our earlier analysis of the metrics led us to conclude it is a background job, and we now know it is one that makes HTTP requests. But we can do better by taking a closer look.

What looked like a straight-line-steady increase is actually stair-stepped, and although there is some noise in the data, it looks like we see a big increase every 1 hour.

It turns out that our coderd service runs 4 background jobs every 1 hour. We have 4 suspects. Which one could it be? Could it be more than one of them? A quick fix to help our sleuthing is to alter the period of the job runs. I could do this one job at a time, but wouldn’t it be nice to make one change that I could leave in place?

The choice of 1 hour intervals is what most people would consider “a nice round number.” But unlike people, computers don’t care about round numbers<a href="#2">[2]</a>. 1 hour isn’t an exact value we care about, so it’s fine to change it to something close and not expect the system-wide behavior to be affected.

It’s best to avoid round numbers, in fact the less round, the better. I choose the most “unround” numbers of them all: prime numbers. That makes it mathematically impossible for the period to be some multiple of a more frequent job so they continually happen at the same time. Choose a base unit, and then make your jobs prime numbers times that base unit. I chose 1 minute as the base unit so it will still be easy to read off a metric chart, and because our Prometheus metrics are only scraped every 15 seconds or so, so going much smaller than 1 minute won’t help correlating to metrics. So, my 1 hour jobs become 47, 53, 59, and 67 minutes.

The spikes in goroutines are now mostly 47 minutes apart, although there is still some noise. This extra noise threw me for a loop, and made me wonder whether more than one job was responsible for the leaks. But, the power of choosing prime numbers means that job starts are staggered, and when looking at the peaks alongside metrics that show when jobs start and stop, the peaks always correlated with either the start or the stop of the same job.

So, I have my prime suspect: a job that checks all our workspace container image tags for updates from their respective registries. It’s got motive: HTTP requests to container image registries. It’s got opportunity: I can correlate it, over and over, to the time of the crime.

<h1>Interlude: Searching</h1>

Above I have described to you the things that I did that actually made progress in my investigation. While doing those things, I was also pounding sand in an activity that did not make progress: reading code.

With a good IDE, you can drill down though call stacks and back up. This is tedious, but I’d like to think I’m pretty good at understanding code and this kind of depth-first search for the buggy code has actually netted me the bug on more than one occasion in the past.

But, as previously mentioned, HTTP is very ubiquitous, so there is a lot of ground to cover, and it’s not always clear when to stop drilling down. Another HTTP request could be just another level down. Alas, it was not to be on this outing.

<h1>Tracking it Down</h1>

The game is afoot! But, there are still many different code paths. Interacting with a container image registry is <a href="https://docs.docker.com/docker-hub/api/latest/">a complex interaction involving many HTTP requests</a>.

I also spend some time getting to know the victim. At first glance, they are all John Doe. We don’t know what was being requested when they died. But, on closer inspection of the transport code, the leaked readLoop goroutine turns out to have a reference to the HTTP request it died trying to return the response for. If only I could read that request! I’d know the URL, and it could crack the case wide open.

I could read it if I had it in a debugger.

I try to reproduce the leak in some hastily constructed unit tests, but no luck. It’s not just any container image from just any registry that results in the leak.

Then it dawns on me: I know exactly the set of container images that reproduce the bug. It’s the one in our dogfood cluster. I clone the database so that my testing won’t disrupt other users or delete any data, then run the background job in my own workspace, with debugger, against the cloned data.

Bingo.

Stepping through the job with a debugger, I can see the leaked goroutine. I can read its memory from beyond the grave. From the request URL, I can tell what image it is. The goroutine is no longer John Doe! But, the deed is done and goroutine leaked. I still need to find the perpetrator.

Armed with knowledge of the container image, I run the job one more time, adding some code at the top level that skips everything but the one I know results in the leak. I also add a breakpoint deep in the HTTP transport RoundTripper, so that I will break on every HTTP request. My trap is set and I lay in wait for the perpetrator. I know I’m close.

I click, click, click ahead on my breakpoint as HTTP requests come in, waiting for the exact URL I know to be the trigger. And then, I have it: the callstack where the bug lies.

It’s just as we suspected: a missing response.Body.Close(). <a href="https://github.com/google/go-containerregistry/pull/1482">The PR with the fix is a one liner</a>. It’s a little poetic, for so much effort, to write a single line of code.

The results speak for themselves.

<h1 id="key-takeaways">Key Takeaways</h1>

While every leaky code is leaky in its own way, many of the leak detective’s techniques and analytical tools are general.

<ol>
<li>Monitor your services for leaks. Key resources to check for every piece of Go software include:
a. Memory
b. Goroutines
c. File descriptors (open files)</li>
<li>Depending on your application, you might also want to monitor:
a. Disk space
b. Inodes
c. Child processes
d. Special resources used by your application (IP Addresses?)</li>
<li>Look at the rate at which resources are leaked.
a. Does the rate correlate to load? Likely related to the request path of your service
b. Is the rate independent of load? Likely a background job</li>
<li>Avoid running all your background jobs on the exact same schedule. Use prime numbers to avoid overlapping job runs.</li>
<li>Use metrics or logs to record background job start and end times; look for correlations between these times and leaks.</li>
<li>If you can, export or clone real data to reproduce the problem in your IDE.</li>
</ol>

On the last point, a word of caution: be careful about cloning production data that includes external customer or user data. Consult with your security team before copying data if you are at all unsure. This is doubly true if you operate in a regulated industry (finance, healthcare, etc.) or are a spy.

<a id="1">[1]</a> Using your own product is colloquially known as dogfooding, as in, “eating your own dog food.”
<a id="2">[2]</a> And, they have a <a href="https://xkcd.com/1000/">different idea</a> of what counts as round.

Last week, an engineer noticed something suspicious in our dogfood cluster’s main coder service, coderd...

AI-native Development

Secure Environments for 2,500+ Developers

Spike Curtis

Explore the latest from Spike

How Changing One Constant Reduced Our CI Memory Usage By 70%

Query-Mutating Data Race in Go: Hiding in Plain Sight

Introducing Quartz: A Deterministic Time Testing Library for Go

How We Improved the Performance of a Userspace TCP Stack in Go by 5X

Go Testing: Contexts and t.Parallel()

HTTP Resource Leak Mysteries in Go

Related content

Using Claude Code with Coder Workspaces

Introducing Coder Agents

Self-Hosted Doesn’t Always Mean What It Implies