Mike Suchacz

Engineering Manager, Coder

<h2>Every flake comes with a cost</h2>

Flaky tests are nothing new - every engineer has experienced them. When deadlines are tight - it&#39;s so much easier to click rerun than dive deep into root cause analysis. The bigger the product - the higher the complexity of the investigation process. At Coder we run thousands of tests across many jobs - most of them involve networking, concurrency and are inherently vulnerable to be flaky. Some stem from OS differences, some from networking intricacies and others are just a mistake somebody made.

Our first step a long time ago was to report flaky tests to our metrics backend and create Slack alerts in our #dev channel. Whenever a merge to main would cause a failing build the engineer responsible would identify the failing test, create, or update an issue, and use git blame to determine an assignee.

In the first half of this year we made a decision to declare war on flakes.

<h2>The boring parts</h2>

Investigating a failed CI run after merge to main is a structured, rule-based process focused on compiling information from multiple sources. The usual process, at a high level, involved: reading logs, checking existing issues for similar occurrences, filling out issues with the new run details and checking git blame. We weren’t doing our best at it - sometimes people forgot, sometimes the check wasn’t as methodical as it could have been.

Solving this process algorithmically is near impossible - the amount of edge cases to cover is almost infinite. At that moment we realized that this could be a perfect job for an AI agent to figure out. However such a bot requires a lot of wiring - it needs GitHub credentials to read job logs and issues, an environment to clone the repo and run git blame, and we need to be able to begin it&#39;s investigation into the flake automatically. Luckily for us, our Research department has recently delivered a little something called Blink.

<h2>One Blink to Bind Them All</h2>

Blink is Coder’s <a href="https://github.com/coder/blink">self-hosted framework</a> for building and deploying AI agents. It connects organizational context and enables communications through Slack. When you first run Blink it will run you through a wizard-like experience of setting up the Slack app and connecting your LLM provider. Once configured, Blink opens a chat interface where you can customize and develop the agentic reasoning through AI assisted coding. Blink automatically handled the infrastructure wiring that normally takes days, giving us a major headstart and chance to focus on deploying the flake investigator to production.

<h2>Our system prompt is a 200-line runbook — and that&#39;s the point</h2>

When we first started building the flake bot, we made the classic mistake: we wrote a prompt that sounded like a job description.

<code>You are a CI flake investigator. When a test fails, analyze the logs, 
determine the root cause, and create a GitHub issue with the right owner.</code>

This produced exactly what you&#39;d expect: confident, plausible-sounding investigations that were wrong in subtle ways. The bot would assign issues to whoever triggered the CI run (almost never the right person). It also reported quite a few false positives.

Our engineers already built a great mental model through dozens of flake investigations. They knew to use the grep command for <code>signal: killed</code> when tests timeout mysteriously. They knew that matrix job cancellations after a rerun are noise, not signal.

This knowledge lived in Slack threads, in code review comments, in the heads of people who&#39;d been burned before.

So we wrote it down. All of it.

1. Failure classification logic

We explicitly enumerate the failure modes the bot should recognize:

<pre><code class="markdown">**Root Cause Classification**
- **A. Flaky Test**: Intermittent failure, timing-dependent
- **B. Data Race**: Race condition detected by Go race detector
- **C. Process Crash**: Multiple &quot;unknown&quot; failures, panic/OOM
- **D. Infrastructure**: Database, network, or CI environment issue
- **E. Code Change**: New failure introduced by recent commit
</code></pre>

This went way beyond documentation. We built an entire decision tree. When the bot sees a failure, it now asks &quot;which of these five categories does this fit?&quot; instead of freestyling an explanation.

2. Detection patterns as literal code

We don&#39;t tell the bot to &quot;look for signs of a panic.&quot; We give it the exact grep commands:

<pre><code class="markdown">**Panic Detection:**
grep -i &quot;panic:&quot; &lt;log_file&gt;
grep -i &quot;runtime error:&quot; &lt;log_file&gt;
grep -A10 -B5 &quot;panic:&quot; &lt;log_file&gt; # Get context around panics

**OOM Detection:**
grep -i &quot;signal: killed&quot; &lt;log_file&gt;
grep -i &quot;cannot allocate memory&quot; &lt;log_file&gt;
</code></pre>

This matters because LLMs are pattern-matchers, and we&#39;re giving them the patterns. When we added the data race detection section—complete with the exact <code>WARNING: DATA RACE</code> string that Go&#39;s race detector emits—our accuracy on race-related flakes jumped immediately.

3. Explicit anti-patterns

Some of our most valuable prompt lines are prohibitions. This section exists because the bot kept making this exact mistake:

<pre><code class="markdown">**NEVER** assign based solely on:
- Commit author of the failing CI run
- PR author that triggered the failure 
- Most recent committer to main branch
</code></pre>

We also learned to call out specific false-positive scenarios:

<pre><code class="markdown">**IGNORE: Matrix Job Cancellation Artifacts**
Do NOT create issues for these scenarios:
- A matrix job fails, causing other jobs to get cancelled
- Even if the rerun passes, a Slack message is still sent due to 
 the cancellation of other matrix jobs in the previous run
</code></pre>

4. Quality checklists

At the end of the prompt, we include an explicit checklist:

<pre><code class="markdown">Quality Checklist
- [ ] Used `git blame` on specific failing test function
- [ ] Searched for data race warnings in race test failures
- [ ] Searched with at least 3 different query patterns
- [ ] Checked last 30 days of closed issues
- [ ] Verified not duplicate of existing issue
- [ ] Assignment based on code ownership, not commit author
</code></pre>

Checklists make the model way less likely to skip steps. The format implies &quot;all of these, every time.&quot;

5. Output templates

We specify exact title formats because consistency matters for searchability:

<pre><code class="markdown">**Title Formats**:
- Single test: `flake: TestName`
- Data race: `flake: Data race in [component] - TestName`
- Process crash: `flake: Test process crash - [cause]`
</code></pre>

When every flake issue follows the same naming convention, duplicate detection actually works.

<h3>Where we ended up?</h3>

We introduced this on October 1st, 2025. Since that the bot has analyzed and identified 113 issues out of which only 7 ended up being reassigned to different owners. 94% correctness ain’t that bad considering the bot itself responded to 95% of incoming requests. 5% of failures were mostly Blink bugs we’ve been fixing along the way (exceeding the context window, slack webhook behavior, etc.).

<h3>The prompt is never done</h3>

Building this runbook taught us what good prompting actually looks like: take implicit knowledge from experts, make it explicit, and refine it through iteration. That&#39;s exactly what we did when moving from &quot;investigate flakes&quot; to a 200-line decision tree with grep commands and quality checklists. The same discipline applies after deployment.

We version control our system prompt and review changes like code. When the bot makes a mistake, we don&#39;t just correct the output—we ask &quot;what instruction would have prevented this?&quot; and add it to the prompt.

Some recent additions:

<ul>
<li>&quot;If you can&#39;t download all the logs from GitHub, even after retrying a few times, you can assume you don&#39;t have the full picture, and should not investigate the failure.&quot; (Added after the bot confidently diagnosed a flake based on partial logs)</li>
<li>&quot;Check the date of the failure. It should be within a few minutes of the Slack message. If it&#39;s not, you probably have the wrong failing job.&quot; (Added after the bot investigated a week-old failure that happened to match a search query)</li>
</ul>

The prompt is a living document of everything that&#39;s ever gone wrong (just like an older codebase covering all the edge cases).

<h3>The Uncomfortable Truth</h3>

There&#39;s nothing magical here. Our &quot;AI-powered flake investigation system&quot; is, at its core, a very long checklist being executed by a language model with tool access.

But that&#39;s precisely what makes it work. The LLM provides the flexibility to handle natural language logs, the judgment to weigh ambiguous evidence, and the tirelessness to run the same investigation at 3 AM that it runs at 10 AM. The prompt provides the structure to make that judgment reliable.

The best LLM prompts aren&#39;t clever. They&#39;re comprehensive. They&#39;re the runbook you always meant to write but never had time for—except now, something actually follows it.

<h2>Additional materials:</h2>

<ul>
<li><a href="https://blink.coder.com/">https://blink.coder.com/</a></li>
<li><a href="https://docs.blink.so/">https://docs.blink.so/</a></li>
</ul>

Flaky tests are nothing new - every engineer has experienced them. Coder runs thousands of tasks that are vulnerable to be flaky. This year, we declared war on flakes to automate the worst parts of CI maintenance. 

AI-native Development

Secure Environments for 2,500+ Developers

Mike Suchacz

Explore the latest from Mike

In a Blink: Automating the worst part of CI maintenance

Related content

Using Claude Code with Coder Workspaces

Introducing Coder Agents

Self-Hosted Doesn’t Always Mean What It Implies