Elevating Developer Experience: Use Coder to Measure and Optimize Developer Productivity

Rob Whiteley

 on January 8th, 2024

Legendary management consultant Peter Drucker once famously said, “You can't manage what you can't measure.” This holds true for software development too. At Coder, we know how important developer experience (DX) is to the success of developers, their teams, and their organizations.

Before you measure what matters, you need to understand what matters to developers. Developer experience is about how developers interact with their tools and environment. Of course, developers want to deliver quality code, on time. Improving DX removes the obstacles that let them do this.

You don’t have to be a developer productivity expert to improve DX. If your organization is lucky enough to have a developer productivity expert, or you are one yourself, then you already know these things. But do you know about Coder? Elevating DX is probably the biggest reason we started Coder and other open source projects.

So let’s look at some opportunities to improve DX and some key metrics that give you insight about them. Along the way, you’ll see how Coder makes elevating and measuring DX an iterative, productive process.

Metric 1: Onboarding time

New developers don’t deliver when they’re onboarding. A lot of teams just accept this as a cost of doing business. That’s an entirely reasonable thing to believe. After all, it takes weeks, or longer, before a new developer is fully productive. And there’s the extra cost to the rest of the team as they support the new developer instead of coding.

You’ll see the effects of onboarding with these metrics:

• Time to first commit to production: From their first day, how long does it take a new developer to commit code that goes to production?

• Velocities of other teammates: You might also notice that a team’s velocity slows down when it takes on a new teammate.

Onboarding might seem like one of the easiest metrics to reduce. It is, as you’ll see in a moment. But efforts to reduce it can go to an extreme. Of course, it takes time to train a new developer on your organization’s technology and processes. And it’s important for a team to take the time to share team culture with the new developer.

But the time spent to repeat the tedious, often undocumented tasks to get a new developer up to speed is time wasted. This is one of Coder’s superhero powers: reproducing development environments at the literal click of a button. A new developer doesn’t have to chase down the IT team for software licensing or special hardware. There’s no need to break the flow of other developers to help sort out dependencies and versions. Coder lets a team set up, document, and track these complexities as a single source of truth then reproduce it effortlessly and instantly, for each team member. Improvements to a template are instantly available to its workspace users. We’ve heard of amazing success stories from our users, even reducing the time to first commit to the first day!

Metric 2: Inner loop optimization

The “inner loop” is the work that developers do in their development environment. Developers refactor and build several times a day, but their IDE often leaves them waiting while it generates indexes or loads sub-projects.

We consider this the missing metric. Despite developers playing with swords, not many teams actually invest in it, let alone measure it.

Metrics for inner loops depend on your team’s workflow. Here are a couple of obvious metrics:

• Time to first file open: How long does a developer wait after opening their IDE before they can start editing a file?

• Build time: How long does it take to build a project?

These inner loop metrics are purely quantitative, and they’re often easy to measure. For example, measuring build time for a Java project might be as simple as running time gradle build on the command line.

Once you’ve identified the slow parts of the inner loop, you can look into speeding them up. Most development tools support incremental builds or caching indexes. Coder is especially useful for leveraging these features. For example, IntelliJ has shared indexes which Coder can configure and generate. For an extra boost, a team can use Coder to cache common assets with faster networking and storage in the cloud instead of waiting for a laptop’s wi-fi connection.

Metric 3: Bug resolution time

Defects cost a lot, of course. They cost testing time to find, reproduce, and report. They steal time from new feature development. And they cost sales. So the cost to prevent defects gives a good return on investment.

There are countless websites, services, and consultants, happy to help you with metrics for testing. We’ll only touch on some simple, but useful, metrics for bug resolution time:

• Time between defect detection and resolution: This is a simplistic metric, to be sure. Still, a team with a decelerating bug resolution time is a sign of trouble.

• Bug severity distribution, bug assignments, bug reporting frequency: These metrics give clues about the quality of your code. A disproportionate number of high-priority bugs is a signal of poor code quality. Or it might be a sign of the quality of the bug reports themselves. Are bugs clearly reported and reproducible? Do they contain sufficient information for a developer to work from?

• Defect resolution efficiency (DRE): To pinpoint which part of your process is causing more defects, consider DRE. It’s a more sophisticated set of metrics than just time between detection and resolution. DRE measures the percentage of defect resolutions between different phases of development. DRE is a good prevention tool: the sooner you find and resolve a defect, the less it costs your organization.

Coder isn’t a defect management tool. But it certainly improves the chances of finding and fixing them. Another one of Coder’s super powers is never again hearing the words “but it works on my machine”. Coder prevents this situation because developers, testers, and deployers all work from identical, consistent environments that use the same code, dependencies, and data. Define it once as a template, use it many times in workspaces that behave identically.

Metric 4: Code reviews

A proven way to improve development outcomes, with some beneficial side effects, is timely, consistent peer code reviews. Code reviews are a vital tool for managing technical debt, preventing defects, improving maintainability, and encouraging a culture of collaboration, professional development, innovation, and problem solving.

The subject of code reviews is a broad one, so we’ll cover some metrics that let you pick some juicy, lower hanging fruit:

• Response time for code reviews: How long does it take before a teammate responds to a PR? Does your team’s workflow allow a developer to work on other issues while waiting for a PR? How do team leads, managers, or product owners manage assignments for code reviews?

• Number of comments per review, ratio of accepted and rejected code changes: These metrics give clues to the quality of the reviews.

Code reviews are more productive when the participants get on the same page then interact directly. Does your team have a code review style guide? When is it appropriate to just type LGTM (Looks good to me)? When does code deserve more in-depth comments?

Repo platforms like GitHub and GitLab have excellent code commenting capabilities. You can complement these features with modules in Coder workspaces to connect developers to repos and their teammates. For example, add modules for cloning repos in GitHub, submit PRs, and starting IDEs. When reviewing, developers are confident they are looking at the same code with the same tools.

Metric 5: Deployment frequency

Increasing deployment frequency is often a big challenge for development teams. I’m sure you’ve heard about horror stories, and even survived a few yourself. But the reasons for more deployments are clear, including faster time to market, happier customers, and risk mitigation for big projects.

There are many causes and effects in and outside your team that affect deployment frequency. Once you’ve decided on a frequency, here are some simple metrics that help you stay on track:

• Deployment frequency: This is an obvious one, but it still helps to understand the nature of your deployments. What is the historical average time to deploy? For example, can your team promise monthly deployments if it averages 12 deployments in a year? Or can it deploy on the 1st of every month for a year?

• Success/failure rate of deployments: How often are deployments rolled back or re-deployed because of an incident? This is another obvious one, but it gives a clue to how well your teams are collaborating and detects early signs of weaknesses.

Internally, deploying more frequently means smoother-running, more collaborative development, testing, and deployment teams. We won’t go into the established methods for increasing deployment frequency, like implementing CI/CD, building cross-functional teams, and so on. Instead, think of Coder as the foundation that you build these things onto. Coder reduces the friction of setting up development environments. It makes it easier and faster for testers, deployers, and other contributors to make changes without waiting on tickets.

Elevate, measure, refine, repeat

There’s a range of metrics for elevating developer experience. Just measuring onboarding time confirms the cost of taking in a new developer. More sophisticated metrics, like DRE, reveal opportunities to improve the entire devops workflow. We made Coder to take advantage of these insights. Coder and its other open source projects turn DX into an iterative activity. You improve your team’s DX with confident decisions. And you get that confidence from consistent, accurate metrics.

Start elevating your team’s developer experience

Get started with Coder to elevate and measure DX. Coder is easy to install with just a script. Sign up for a 30-day trial of Coder Enterprise for enhanced collaboration, security, governance, and management features.