Skai boosts development with Cloud Development Environments

Skai is an omnichannel advertising platform, uniquely enabling some of the largest brands and agencies in the world to run data-driven programs across walled garden media. It empowers both media leaders and activation teams to drive impactful results from their advertising program.

In this blog you will learn how Skai’s Engineering has:

The Engineering

As Skai’s mission requires large-scale advertisement data retrieval and processing and runs on a complex and multi-generational tech stack. In the middle are 350+ engineers developing publisher integrations, data processing, AI-based insight generation, dashboards and grid layouts for our platform to display.

To sustain all of that, Skai’s stack – running on AWS – is composed of hundreds of Microservices, Serverless apps, Airflow Dags plus a 3M LoC monolith in its center.

Developing on the Monolith

One of the biggest pain points Skai engineering faced is developing on its Java based monolith.

• Average onboarding time of up to 5 days. This includes both new hires and seasoned engineers who need to retool.
• Working on the project in the IDE was a challenge. It was not uncommon for the IDE to drastically slow down or even crash with Out of Memory errors when indexing and building.
• Developers would need to reauthenticate with Skai’s various cloud systems multiple times a day, which often caused confusion and errors in the process of building and running the environment.
• Gradle based scripts created to manage the local components were inconsistent, requiring multiple manual commands to start various flavors of our environment locally from the IDE.

The issues were further enhanced by multiple operating systems, CPU-architectures and package versions installed on developer laptops. That often meant that solving one issue would solve it for one group of developers, while keeping it unsolved for the rest. It was a nightmare!

CDEs to the Rescue

At this point, Skai’s infra engineering team realized they needed a standard, templated and reproducible development experience. In came the idea of a Cloud Development Environment (CDE). CDE’s were a relatively new concept, and the team was skeptical about their ability to reproduce such a complex environment as Skai’s monolith while still keeping a low-latency smooth experience for developers.

The team sought a solution with the following requirements:

• Streamlined developer experience of managing CDE lifecycle: Creation, auto-scheling and auto-shutdown.
• Strong integration with Skai’s preferred IDE (IntelliJ IDEA) including IntelliJ Gateway extension support.
• Private networking with Skai’s AWS environment.
• Templated images with Infrastructure as Code.
• Cost effective: Lowest unit cost per developer possible, with the ability to monitor and control costs effectively.
• Security: a dedicated cloud based setup, to adhere to Skai’s enterprise security requirements.

Next, going into the discovery and proof of concept stages, Skai’s infra engineering team tried a few leading CDE managed services. Some felt still in early stages, others lacked the crucial AWS private networking support and some were over budget while others didn’t meet Skai’s security requirements.

Welcome Coder

Finally, the team decided on Coder. It rang all the bells: The beautiful and simple UI, the IntelliJ IDEA coder extension, ability to self-host on Skai’s AWS private environment, the built-in scheduling capabilities and the affordable bring your own cloud hardware model. The team liked the open-sourced nature of Coder, including the Terraform-based management of templates.

The setup was quite simple for the Infra Engineering team to quickly get started:

1. Coder single-pod deployment on its AWS EKS cluster and an AWS RDS instance for the Coder control plane,
2. User authentication with Skai’s GitHub cloud Enterprise organization.
3. AWS EC2 based instances for developer workspaces, first based in Skai main region North Virginia, later expanded to further regions.
4. Networking setup required setup of IPSEC tunnels from Skai’s VPN cloud provider to relevant AWS accounts, in order to create a low-latency network experience for developers.
5. Terraform based workspace template for the first use case: the monolith.

Leveling the Playing Field

Skai’s infra engineering team set out to create a standardized and smooth experience for our developers. The template creation process for the first Monolith use case involved investment in multiple pain areas such as:

• Adding metric reporting to the local environment for improved observability.
• Developing a complete and simple Gradle entry-point wrapper to allow developers to initialize their environment with a single command.
• Standardizing the way developers authenticate with AWS resources.
• Adding Gradle remote cache to improve build time and IntelliJ IDEA branch indexing performance.

The outcome was one standard image that a developer can easily launch and get the same experience every time. And if something goes wrong - they could just throw the old environment away and lunch a fresh one.

The Results: A whole new world!

Once the Skai team had a template ready it was time to roll it out to the organization. Since CDE’swere a completely new concept for Skai’s engineering org, adoption was totally voluntary, allowing developers to choose if they want to continue working on their local laptop-based development environments, switch completely to Coder CDE’s or use a hybrid solution. The assumption was - if it's to give the users good value they adoption will come.

And boy, did it come.

In a matter of 6 months the adoption grew consistently from a few users to 60 Daily Active Users, and a total of 200+ developers using the template.

The results were highly positive:

1. Developers reported launching a new environment took minutes instead of days!
2. Devs boosted their productivity by launching a few environments to work on several features in parallel or use it for code reviews.
3. The boot time of Skai’s monolith environment was cut down by 10 minutes due to more modern cloud hardware.
4. CDE’s rapidly support Skai’s strategic decision to expand its development teams to other regions of the world. A new region would take only a couple of days to launch and very quickly Skai’s Coder deployment grew globally to 3 different regions.
5. Quick expansion of above benefits into other types of environments: Apache Airflow DAG development. Python Microservices, Java Microservices and more.

The solution is not by any means perfect, there are still some issues around IntelliJ Gateway and version compatibility but overall, CDE’s and Coder have revolutionized the way Skai Engineering is working with local development and its ability to more quickly release product features and rapidly scale the org.